Video filling method, device, equipment and storage medium
By aligning and fusion of the features of the image frames to be filled, adjacent image frames and loop feedback frames in the video, problems such as picture jitter in video filling are solved, and the video filling effect and visual consistency are improved.
Patent Information
- Application Number
- CN202010613541.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-06-30
- Publication Date
- 2024-11-08
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing image filling technology causes problems such as picture shaking, flickering, and blurring in video filling processing, and the video filling effect is poor.
By acquiring the characteristics of the image frame to be filled in the video, its adjacent image frames and loop feedback frames, it is aligned and fused, and the combined processing is combined to reconstruct the missing area, form the complete image frame, and use it as the next image frame to be filled in. The loop feedback frame is used to realize the effective utilization of time domain information by using the alignment and fusion of multi-frame image frames.
Improve the video filling effect, reduce the picture quality difference after video filling, and enhance the visual consistency of the video.
Smart Images

Figure CN111738913B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the fields of artificial intelligence, deep learning, computer vision, and image filling technology, and specifically, to a video filling method, apparatus, device, and storage medium. Background Art
[0002] Image filling technology has a wide range of application scenarios, such as image editing, image restoration, removing specific objects in images, etc. Most existing image filling technologies are based on block matching or texture matching methods, using ordinary convolutional neural network processing. Applying image filling technology to the filling process of videos means processing multiple video frames of video files frame by frame, and each video frame is processed independently of each other, which makes the overall video file in the later stage prone to problems such as jitter, flickering, blur, etc., and the video filling effect is unsatisfactory. Summary of the invention
[0003] The present application provides a video filling method, apparatus, device and storage medium.
[0004] According to a first aspect of the present application, a video filling method is provided, comprising:
[0005] Acquire an image frame to be filled in a video, a plurality of adjacent image frames of the image frame to be filled, and at least one loop feedback frame, wherein the plurality of adjacent image frames include a plurality of image frames before the image frame to be filled and a plurality of image frames after the image frame to be filled, the image frame to be filled and the plurality of adjacent image frames include missing regions, and the loop feedback frame does not have missing regions;
[0006] Extracting features of the to-be-filled image frame, the plurality of adjacent image frames, and the loop feedback frame respectively;
[0007] Aligning the features of the plurality of adjacent image frames and the features of the loop feedback frame with the features of the image frame to be filled, respectively, and fusing the aligned features to obtain fused features;
[0008] Combining the features of the image frame to be filled with the fusion features to obtain a combined feature;
[0009] An image frame is reconstructed based on the combined features, and there is no missing area in the reconstructed image frame; wherein the image frame reconstructed this time is used as a loop feedback frame in the filling process of the next image frame to be filled.
[0010] According to a second aspect of the present application, a video filling device is provided, comprising:
[0011] An image frame acquisition module, used to acquire an image frame to be filled in a video, a plurality of adjacent image frames of the image frame to be filled, and at least one loop feedback frame, wherein the plurality of adjacent image frames include a plurality of image frames before the image frame to be filled and a plurality of image frames after the image frame to be filled, the image frame to be filled and the plurality of adjacent image frames include missing regions, and the loop feedback frame does not have missing regions;
[0012] A feature extraction module, used to extract features of the to-be-filled image frame, the plurality of adjacent image frames and the loop feedback frame respectively;
[0013] A feature alignment and fusion module, used to align the features of the plurality of adjacent image frames and the features of the loop feedback frame with the features of the image frame to be filled, and fuse the aligned features to obtain a fused feature;
[0014] A feature combination module, used for combining the feature of the image frame to be filled with the fusion feature to obtain a combined feature;
[0015] An image frame reconstruction module is used to reconstruct an image frame based on the combined features, wherein there is no missing area in the reconstructed image frame; wherein the image frame reconstructed this time is used as a loop feedback frame in the filling process of the next image frame to be filled.
[0016] According to a third aspect of the present application, an electronic device is provided, including:
[0017] at least one processor; and
[0018] a memory communicatively connected to at least one processor; wherein,
[0019] The memory stores instructions that can be executed by at least one processor. The instructions are executed by the at least one processor so that the at least one processor can perform the method described above.
[0020] According to a fourth aspect of the present application, a non-transitory computer-readable storage medium storing computer instructions is provided, where the computer instructions are used to cause a computer to execute the method as described above.
[0021] According to a fifth aspect of the present application, a computer program product is provided, comprising a computer program, wherein the computer program implements the method as described above when executed by a processor.
[0022] The embodiments of the present application take into account the characteristics of video filling. By aligning and fusing multiple image frames, the contents of adjacent different frames can be complementary, thereby achieving effective use of the time domain information of the video. In applications, the effect of video filling can be improved, and the problem of poor picture quality after video filling can be reduced to a certain extent.
[0023] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present application.
[0025] Figure 1 is a flowchart of a video filling method according to an embodiment of the present application;
[0026] Figure 2 is a structural block diagram of a video filling device according to an embodiment of the present application;
[0027] Figure 3 is a schematic image to be filled in one embodiment of the present application;
[0028] Figure 4 is a logic process diagram of a video filling method according to another embodiment of the present application;
[0029] Figure 5 Yes Figure 3 Schematic image obtained after image filling processing in the embodiment;
[0030] Figure 6 It is a block diagram of an electronic device for implementing the video filling method of an embodiment of the present application. DETAILED DESCRIPTION
[0031] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.
[0032] In order to more clearly describe the embodiments of the present application, the relevant contents of semantic segmentation, instance segmentation and mask processing of images in the field of deep learning are first briefly described.
[0033] The embodiment of the present application is based on the instance segmentation information of the image. Figure 1 The flowchart of the video filling method according to the embodiment of the present application is shown, and includes the following steps:
[0034] S101: Acquire an image frame to be filled I_t in a video, a plurality of adjacent image frames of the image frame to be filled I_t, and at least one loop feedback frame I'_{t-1}, wherein the plurality of adjacent image frames include a plurality of image frames before the image frame to be filled I_t and a plurality of image frames after the image frame to be filled I_t, the image frame to be filled I_t and the plurality of adjacent image frames include missing regions, and there is no missing region in the loop feedback frame I'_{t-1};
[0035] S102: extracting features of the image frame to be filled I_t, multiple adjacent image frames and the loop feedback frame I'_{t-1} respectively;
[0036] S103: aligning features f_s of multiple adjacent image frames and features f'_{t-1} of the loop feedback frame with features f_r of the image frame to be filled, and fusing the aligned features to obtain a fused feature f_s';
[0037] S104: Combining the feature f_r of the image frame to be filled with the fusion feature f_s' to obtain a combined feature f_c;
[0038] S105: reconstructing the image frame o_t based on the combined feature f_c, wherein there is no missing area in the reconstructed image frame; wherein the image frame reconstructed this time is used as a loop feedback frame in the filling process of the next image frame to be filled.
[0039] An embodiment of the present application is to fill a video with missing areas (for example, a damaged image), and the image frame I_i to be filled is a missing image frame in the video. The video of the embodiment of the present application contains multiple frames of image frames I_i to be filled, and the missing parts of the image frames are the areas to be filled.
[0040] According to an embodiment of the present application, for a video file containing a missing area, the video file is not directly filled using a neural network model, but firstly, an image frame to be filled I_t, multiple adjacent image frames of the image frame to be filled I_t, and at least one loop feedback frame I'_{t-1} in the video need to be obtained, wherein the multiple adjacent image frames include multiple image frames before the image frame to be filled I_t and multiple image frames after the image frame to be filled I_t. For example, schematically, N / 2 image frames I_{tN / 2} before the filled image frame I_t and N / 2 image frames I_{t+N / 2} after the image frame to be filled I_t can be taken, and a total of N adjacent image frames can be obtained for subsequent processing;
[0041] Among them, the addition of the loop feedback frame I'_{t-1} is to form a loop feedback mechanism designed in the embodiment of the present application. The loop feedback frame I'_{t-1} can be a complete image frame obtained after the previous filling process on the video. Since the complete image frame carries the image information after the missing area is filled, adding the complete image frame to the processing of the next frame of the image can enhance the filling effect.
[0042] Then, the features of the image frame to be filled I_t, multiple adjacent image frames and the loop feedback frame I'_{t-1} are extracted respectively, and the obtained features can be expressed as f_r, f_s, f'_{t-1} respectively; then, the features f_s of multiple adjacent image frames and the features f'_{t-1} of the loop feedback frame are aligned with the features f_r of the image frame to be filled respectively, and the aligned features are fused to obtain the fused features f_s'. Here, since multiple adjacent image frames contain the time domain information in the video, aligning and fusing them can realize the effective use of the time domain information of the video, and the contents between adjacent different frames can complement each other, which is conducive to improving the video filling effect.
[0043] Afterwards, the feature f_r of the image frame to be filled is combined with the fused feature f_s' to obtain the combined feature f_c; then the image frame o_t is reconstructed based on the combined feature f_c. The reconstructed image frame has been filled according to the fused feature and there is no missing area. The reconstructed image frame can be used as the loop feedback frame of the next image frame to be filled, realizing the continuous application of the loop feedback mechanism.
[0044] It can be seen that the embodiments of the present application take into account the characteristics of video filling, and through the alignment and fusion of multiple image frames, the contents of adjacent different frames can be complementary, thereby achieving effective use of the time domain information of the video and improving the effect of video filling in the application.
[0045] In one embodiment of the present application, the features of multiple adjacent image frames and the features of the loop feedback frame are respectively aligned with the features of the image frame to be filled, and the aligned features are fused, which can be achieved by the following steps:
[0046] Based on the optical flow features of the objects in each image frame, the spatial positions of the same object features in each image frame are aligned;
[0047] The features of each aligned image frame are spliced in the channel dimension to obtain the spliced features.
[0048] Perform feature transformation on the concatenated features to obtain fused features f_s', where the number of fused features after feature transformation in the channel dimension is a preset number.
[0049] Through the above processing, the spatial positions of objects in each frame are aligned based on the optical flow features, for example, by translating up and down in parallel, or translating left and right, and each frame is adjusted so that the spatial coordinates of the same object in each frame coincide with each other.
[0050] In one embodiment, the convolution layer of the convolutional neural network model can be used to perform feature transformation on the aligned feature layer. After the transformation, the length and width remain unchanged, the number of channel dimensions can be changed to a preset number, the transformed features have stronger expression capabilities, redundant features can be removed, and are more conducive to filling in image frames.
[0051] In one implementation, the feature f_r of the image frame to be filled and the fused feature f_s' are combined and processed, which can be achieved by the following steps:
[0052] The combined mask m is calculated based on the absolute value of the difference between the feature f_r of the image frame to be filled and the fused feature f_s';
[0053] The combined feature f_c is calculated according to the following formula:
[0054] Combined feature f_c = (1-m) x f_r + m x f_s'.
[0055] Combining the aligned and fused features with the features of the image frame to be filled is to combine the features of the frame to be filled with the aforementioned aligned and fused features, which can effectively utilize the time domain information in the video, effectively complement the visual content between different frames, enhance the spatial visual consistency, and improve the quality of video filling.
[0056] Corresponding to the above method embodiment, the present application embodiment also provides a video filling device 100, referring to Figure 2 ,include:
[0057] The image frame acquisition module 101 is used to acquire an image frame to be filled I_t in a video, a plurality of adjacent image frames of the image frame to be filled I_t, and at least one loop feedback frame I'_{t-1}, wherein the image frame to be filled I_t includes a missing area, and the plurality of adjacent image frames include a plurality of image frames before the image frame to be filled I_t and a plurality of image frames after the image frame to be filled I_t.
[0058] A feature extraction module 102 is used to extract features of the image frame to be filled I_t, multiple adjacent image frames and the loop feedback frame I'_{t-1} respectively;
[0059] A feature alignment and fusion module 103 is used to align features f_s of multiple adjacent image frames and features f'_{t-1} of the loop feedback frame with features f_r of the image frame to be filled, and fuse the aligned features to obtain a fused feature f_s';
[0060] A feature combination module 104 is used to combine the feature f_r of the image frame to be filled with the fusion feature f_s' to obtain a combined feature f_c;
[0061] The image frame reconstruction module 105 is used to reconstruct the image frame o_t based on the combined feature f_c, and there is no missing area in the reconstructed image frame; wherein the image frame reconstructed this time is used as a loop feedback frame in the filling process of the next image frame to be filled.
[0062] In one embodiment, the feature alignment fusion module includes:
[0063] An alignment submodule, for aligning the spatial positions of features of the same object in each image frame based on the optical flow features of the object in each image frame;
[0064] The stitching module is used to stitch the features of each aligned image frame in the channel dimension to obtain the stitching features.
[0065] The feature transformation submodule is used to perform feature transformation on the spliced features to obtain fused features f_s', wherein the number of fused features after feature transformation in the channel dimension is a preset number.
[0066] In one embodiment, the feature combination module includes:
[0067] The first calculation submodule is used to calculate the combined mask m according to the absolute value of the difference between the feature f_r of the image frame to be filled and the fusion feature f_s';
[0068] The second calculation submodule is used to calculate the combined feature f_c according to the following formula:
[0069] Combined feature f_c = (1-m) × f_r + m × f_s',
[0070] Among them, f_c represents the combined feature, f_r represents the feature of the image frame to be filled, f_s' represents the fusion feature, and m represents the combined mask
[0071] In one embodiment, the video filling device 100 further includes:
[0072] The preprocessing module is used to remove the target area in the target image frame in the video to obtain the image frame to be filled, and the missing area in the image frame to be filled is the area to be filled.
[0073] In one embodiment, the feature extraction module, the feature alignment and fusion module, the feature combination module and the image frame reconstruction module are generated by respectively training a neural network (such as a convolutional neural network, a recurrent neural network or other types of neural networks) using training data, and the training data includes: a real video and a video obtained by erasing objects in the image frames in the real video.
[0074] After the training is completed, the various neural network models or modules can be logically connected to obtain the video filling device 100. When in use, the video to be filled is input into the video filling device, and after being processed by various modules, the filled image frames can be output. Multiple cycles of processing can obtain multiple complete image frames, and a complete video file can be formed.
[0075] The functions of the various modules in the video filling device 100 of the embodiment of the present application correspond to the contents described in the aforementioned method embodiment, that is, the various modules in the video filling device 100 can be used to implement the various steps of the aforementioned video filling method, and the same beneficial technical effects can be obtained. The implementation method of the embodiment of the present application is described in detail below through specific embodiments.
[0076] refer to Figure 3 , schematically shows an image frame I_t to be filled in an embodiment of the present application. For the convenience of description, Figure 3 In the picture, mark the object you want to remove by coloring it, which is a pair of deck chairs on the beach.
[0077] According to the video filling method of the embodiment of the present application, optionally, the preprocessing module can preprocess the target image: remove the target area in the target image frame in the video to obtain the image frame to be filled, and the missing area in the image frame to be filled is the area to be filled.
[0078] exist Figure 3 In an embodiment, the target original image is a beach picture containing a lounge chair. The lounge chair can be removed from the original picture by using known image segmentation techniques (such as edge segmentation and semantic segmentation), thereby obtaining a missing image to be filled, that is, a beach image with a missing lounge chair area, and the missing area is the area to be filled.
[0079] Figure 4 The logic process diagram of the video filling method of the embodiment of the present application is schematically shown. Figure 4From the perspective of image frame I_t to be filled, the 4 adjacent image frames before and after the image frame I_t to be filled, and the complete image frame o_{t-1} after filling the previous frame (as a loop feedback frame) are respectively input into the feature extraction module for feature extraction to obtain the features of each image frame (which can be expressed as f_r, f_s, f'_{t-1} respectively), and then enter the feature alignment and fusion module for alignment and fusion processing. After the fused feature f_s' is output, it is processed with the image frame I_t to be filled into the feature combination module to obtain the combined feature f_c. Finally, the frame reconstruction module is used to reconstruct the image frame with the combined feature f_c to obtain the output complete image frame o_t.
[0080] Figure 5 The reconstructed image frame is schematically shown, and it can be seen that Figure 5 The filling of the area corresponding to the deck chair in the image frame incorporates the characteristics of the beach and the reflection of the tree. Similar filling processing is performed on the missing image frames in the video to obtain a fully filled video file.
[0081] During the video filling process of the embodiment of the present application, the aligned and fused features are combined with the features of the image frame to be filled, which can effectively utilize the time domain information in the video, effectively complement the visual content between different frames, enhance the spatial visual consistency, and improve the quality of video filling.
[0082] According to an embodiment of the present application, the present application also provides an electronic device, a readable storage medium and a computer program product.
[0083] like Figure 6 , is a block diagram of an electronic device according to a method of an embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.
[0084] like Figure 6As shown, the electronic device includes: one or more processors 1001, a memory 1002, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display graphical information of a graphical user interface (GUI) on an external input / output device (such as a display device coupled to the interface). In other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 6 A processor 1001 is taken as an example.
[0085] The memory 1002 is a non-transient computer-readable storage medium provided in the present application. The memory stores instructions executable by at least one processor to enable at least one processor to perform the method provided in the present application. The non-transient computer-readable storage medium of the present application stores computer instructions, which are used to enable a computer to perform the method provided in the present application.
[0086] The memory 1002 is a non-transient computer-readable storage medium that can be used to store non-transient software programs, non-transient computer executable programs and modules, such as program instructions / modules corresponding to the methods in the embodiments of the present application. The processor 1001 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 1002, that is, implementing the methods in the above method embodiments.
[0087] The memory 1002 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created based on the use of the electronic device for analyzing and processing search results, etc. In addition, the memory 1002 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 1002 may optionally include a memory remotely arranged relative to the processor 1001, and these remote memories may be connected to the electronic device for analyzing and processing search results via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0088] The electronic device corresponding to the method of the embodiment of the present application may also include: an input device 1003 and an output device 1004. The processor 1001, the memory 1002, the input device 1003 and the output device 1004 may be connected via a bus or other means. Figure 6 In the embodiment, the connection via a bus is taken as an example.
[0089] The input device 1003 can receive input digital or character information, and generate key signal input related to user settings and function control of the electronic device for analyzing and processing search results, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick and other input devices. The output device 1004 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display and a plasma display. In some embodiments, the display device may be a touch screen.
[0090] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, application specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0091] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for programmable processors and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or means (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.
[0092] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (Cathode Ray Tube) or LCD (Liquid Crystal Display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0093] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: Local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.
[0094] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short).
[0095] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution disclosed in this application can be achieved, and this document is not limited here.
[0096] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.
Claims
1. A video filling method, include: Acquire an image frame to be filled in a video, a plurality of adjacent image frames of the image frame to be filled, and at least one loop feedback frame, wherein the plurality of adjacent image frames include a plurality of image frames before the image frame to be filled and a plurality of image frames after the image frame to be filled, the image frame to be filled and the plurality of adjacent image frames include missing regions, and the loop feedback frame does not have missing regions; Extracting features of the to-be-filled image frame, the plurality of adjacent image frames, and the loop feedback frame respectively; Aligning the features of the plurality of adjacent image frames and the features of the loop feedback frame with the features of the image frame to be filled, respectively, and fusing the aligned features to obtain fused features; Combining the features of the image frame to be filled with the fusion features to obtain a combined feature; Reconstructing an image frame based on the combined features, where there is no missing area in the reconstructed image frame; wherein the image frame reconstructed this time is used as a loop feedback frame in the filling process of the next image frame to be filled; The step of aligning the features of the plurality of adjacent image frames and the features of the loop feedback frame with the features of the image frame to be filled, and fusing the aligned features, comprises: Based on the optical flow features of the objects in each image frame, the spatial positions of the same object features in each image frame are aligned; The features of each aligned image frame are spliced in the channel dimension to obtain the spliced features. Performing feature transformation on the splicing features to obtain the fused features, wherein the number of the fused features after feature transformation in the channel dimension is a preset number.
2. The method according to claim 1, in, The combining the feature of the image frame to be filled with the fusion feature comprises: Calculating a combined mask according to the absolute value of the difference between the feature of the image frame to be filled and the fused feature; The combined feature is calculated according to the following formula: Combined feature f_c = (1-m) × f_r + m × f_s' Among them, f_c represents the combined feature, f_r represents the feature of the image frame to be filled, f_s' represents the fusion feature, and m represents the combined mask.
3. The method according to claim 1, further comprising: include: The target area in the target image frame in the video is removed to obtain the image frame to be filled, and the missing area in the image frame to be filled is the area to be filled.
4. The method according to any one of claims 1 to 3, in, The steps of acquiring an image frame to be filled in a video, a plurality of adjacent image frames of the image frame to be filled, and at least one loop feedback frame, respectively extracting features of the image frame to be filled, the plurality of adjacent image frames, and the loop feedback frame, respectively aligning the features of the plurality of adjacent image frames and the features of the loop feedback frame with the features of the image frame to be filled and fusing the aligned features, combining the features of the image frame to be filled with the fused features, and reconstructing the image frame based on the combined features are respectively performed by a feature extraction module, a feature alignment and fusion module, a feature combination module, and an image frame reconstruction module, in, The feature extraction module, the feature alignment and fusion module, the feature combination module and the image frame reconstruction module are generated by training the neural network using training data respectively, and the training data includes: real video and video obtained by erasing objects in the image frame of the real video.
5. A video filling device, include: An image frame acquisition module, used to acquire an image frame to be filled in a video, a plurality of adjacent image frames of the image frame to be filled, and at least one loop feedback frame, wherein the plurality of adjacent image frames include a plurality of image frames before the image frame to be filled and a plurality of image frames after the image frame to be filled, the image frame to be filled and the plurality of adjacent image frames include missing regions, and the loop feedback frame does not have missing regions; A feature extraction module, used to extract features of the to-be-filled image frame, the plurality of adjacent image frames and the loop feedback frame respectively; A feature alignment and fusion module, used to align the features of the plurality of adjacent image frames and the features of the loop feedback frame with the features of the image frame to be filled, and fuse the aligned features to obtain a fused feature; A feature combination module, used for combining the feature of the image frame to be filled with the fusion feature to obtain a combined feature; An image frame reconstruction module is used to reconstruct an image frame based on the combined features, wherein there is no missing area in the reconstructed image frame; wherein the image frame reconstructed this time is used as a loop feedback frame in the filling process of the next image frame to be filled; The feature alignment and fusion module comprises: An alignment submodule, for aligning the spatial positions of the same object features in each image frame based on the optical flow features of the object in each image frame; The stitching module is used to stitch the features of each aligned image frame in the channel dimension to obtain the stitching features. The feature transformation submodule is used to perform feature transformation on the spliced features to obtain the fused features, wherein the number of the fused features after feature transformation in the channel dimension is a preset number.
6. The device according to claim 5, in, The feature combination module comprises: A first calculation submodule, configured to calculate a combined mask according to an absolute value of a difference between a feature of the image frame to be filled and the fused feature; The second calculation submodule is used to calculate the combined feature according to the following formula: Combined feature f_c = (1-m) × f_r + m × f_s' Among them, f_c represents the combined feature, f_r represents the feature of the image frame to be filled, f_s' represents the fusion feature, and m represents the combined mask.
7. The device according to any one of claims 5, further comprising: include: The preprocessing module is used to remove the target area in the target image frame in the video to obtain the image frame to be filled, and the missing area in the image frame to be filled is the area to be filled.
8. The device according to any one of claims 5 to 7, in, The feature extraction module, the feature alignment and fusion module, the feature combination module and the image frame reconstruction module are generated by training the neural network using training data respectively, and the training data includes: real video and video obtained by erasing objects in the image frame of the real video.
9. An electronic device, It is characterized in that include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 5.
10. A non-transitory computer-readable storage medium storing computer instructions, It is characterized in that The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 5.
