Video generation method, device, electronic device, storage medium and program product

Abstract

The present disclosure provides a video generation method, device, electronic device, storage medium and program product, which relate to the field of artificial intelligence technology, especially to the technical fields of computer vision, augmented reality, virtual reality, deep learning, etc., and can be applied to the metaverse, artificial intelligence-based content generation and other scenarios. The specific implementation scheme is: perform motion capture on the target object in the original video frame to generate motion information of the virtual object; match the motion information of the virtual object with the motion information of the target object; perform background repair on the area where the target object in the original video frame is located to obtain a background video frame; obtain a target video frame based on the motion information of the virtual object and the background video frame.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to technical fields such as computer vision, augmented reality, virtual reality, and deep learning. It can be applied to scenarios such as the metaverse and artificial intelligence-based content generation, and in particular to video generation methods, devices, electronic devices, storage media, and program products. Background Art

[0002] Computer vision technology is a science that studies how to use computers to "see". Computer vision technology can be applied to scenarios such as image recognition, image semantic understanding, image retrieval, 3D object reconstruction, virtual reality, simultaneous positioning and map construction. For each scenario, how to use computer vision technology to generate reasonable and accurate results is worth exploring. Summary of the invention

[0003] The present disclosure provides a video generation method, device, electronic device, storage medium and program product.

[0004] According to one aspect of the present disclosure, a video processing method is provided, comprising: performing motion capture on a target object in an original video frame to generate motion information of a virtual object, wherein the motion information of the virtual object matches the motion information of the target object; performing background repair on the area where the target object in the original video frame is located to obtain a background video frame; and obtaining a target video frame based on the motion information of the virtual object and the background video frame.

[0005] According to another aspect of the present disclosure, a video processing device is provided, including: a capture module, used to capture the motion of a target object in an original video frame to generate motion information of a virtual object, wherein the motion information of the virtual object matches the motion information of the target object; a repair module, used to perform background repair on the area where the target object is located in the original video frame to obtain a background video frame; and a synthesis module, used to obtain a target video frame based on the motion information of the virtual object and the background video frame.

[0006] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a method as disclosed in the present disclosure.

[0007] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method of the present disclosure.

[0008] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the method of the present disclosure is implemented.

[0009] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0011] Figure 1A An exemplary system architecture to which the video generation method and apparatus according to an embodiment of the present disclosure can be applied is schematically shown;

[0012] Figure 1B A schematic diagram schematically shows a scene of a video generation method according to an embodiment of the present disclosure;

[0013] Figure 2 The flowchart of the video generation method according to the embodiment of the present disclosure is schematically shown;

[0014] Figure 3 A schematic diagram schematically shows a modified abnormal mask map according to an embodiment of the present disclosure;

[0015] Figure 4 A flowchart of generating a background video frame according to an embodiment of the present disclosure is schematically shown;

[0016] Figure 5 A schematic diagram of generating a background video frame according to an embodiment of the present disclosure is schematically shown;

[0017] Figure 6 The structural diagram of the image repair model according to the embodiment of the present disclosure is schematically shown;

[0018] Figure 7 A flowchart of a video generation method according to another embodiment of the present disclosure is schematically shown;

[0019] Figure 8 A block diagram schematically shows a video generating device according to an embodiment of the present disclosure; and

[0020] Fig. 9 A block diagram of an electronic device suitable for implementing a video generating method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0021] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0022] The present disclosure provides a video generation method, device, electronic device, storage medium and program product.

[0023] According to an embodiment of the present disclosure, a video generation method is provided, comprising: performing motion capture on a target object in an original video frame to generate motion information of a virtual object. The motion information of the virtual object matches the motion information of the target object. Performing background restoration on the area where the target object is located in the original video frame to obtain a background video frame. Based on the motion information of the virtual object and the background video frame, a target video frame is obtained.

[0024] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision, disclosure and application of user personal information involved comply with the provisions of relevant laws and regulations, take necessary confidentiality measures, and do not violate public order and good morals.

[0025] In the technical solution of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0026] Figure 1A An exemplary system architecture to which the video generation method and apparatus according to an embodiment of the present disclosure can be applied is schematically shown.

[0027] It should be noted that Figure 1A The examples shown are only examples of system architectures to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but do not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios. For example, in another embodiment, the exemplary system architecture to which the video generation method and apparatus can be applied may include a terminal device, but the terminal device may implement the video generation method and apparatus provided by the embodiments of the present disclosure without interacting with the server.

[0028] like Figure 1A As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0029] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients and / or social platform software, etc. (only as examples).

[0030] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0031] The server 105 may be a server that provides various services, such as a background management server (only an example) that provides support for the content browsed by the user using the terminal devices 101, 102, and 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0032] It should be noted that the video generation method provided in the embodiment of the present disclosure can generally be executed by the terminal device 101, 102, or 103. Accordingly, the video generation apparatus provided in the embodiment of the present disclosure can also be set in the terminal device 101, 102, or 103.

[0033] Alternatively, the video generation method provided in the embodiment of the present disclosure may also be generally performed by the server 105. Accordingly, the video generation device provided in the embodiment of the present disclosure may generally be arranged in the server 105. The video generation method provided in the embodiment of the present disclosure may also be performed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the video generation device provided in the embodiment of the present disclosure may also be arranged in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0034] For example, the terminal devices 101, 102, and 103 can collect the original video of the target object dancing, and then send the original video to the server 105, and the server 105 performs motion capture on the target object in the original video frame to generate motion information of the virtual object. The background of the area where the target object is located in the original video frame is repaired to obtain a background video frame. Based on the motion information of the virtual object and the background video frame, a target video frame is obtained. And the target video generated by the target video frame is sent to the terminal device. Alternatively, a server or server cluster that can communicate with the terminal devices 101, 102, 103 and / or the server 105 analyzes the original video and finally obtains the target video.

[0035] It should be understood that Figure 1A The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0036] Figure 1B A schematic diagram of a scene of a video generation method according to an embodiment of the present disclosure is schematically shown.

[0037] like Figure 1B As shown, in the original video frame P110, the target object O111 is dancing in front of a building. The processing operation of the video generation method provided by the embodiment of the present disclosure can be performed on the original video frame P110. A virtual object O121 simulating the action of the target object O111 is generated, and the target object O111 in the original video frame is replaced with the virtual object O121 to obtain a target video frame P120 in which the virtual object O121 is dancing in front of a building.

[0038] The video generation method of the disclosed embodiment provides a technology for transforming the actions of actual target objects in the real world into computer-generated virtual object actions, achieving the effect of dynamic virtual objects performing or displaying in real scenes, and the content of the target video is novel and rich. It can be widely used in games, entertainment, education, medical and other fields.

[0039] It should be noted that the sequence numbers of the operations in the following method are only used as representations of the operations for the purpose of description, and should not be regarded as representing the execution order of the operations. Unless explicitly stated, the method does not need to be executed completely in the order shown.

[0040] Figure 2 The flowchart of the video generation method according to the embodiment of the present disclosure is schematically shown.

[0041] like Figure 2 As shown, the method includes operations S210 to S230.

[0042] In operation S210, motion capture is performed on a target object in an original video frame to generate motion information of a virtual object.

[0043] In operation S220, background restoration is performed on the area where the target object is located in the original video frame to obtain a background video frame.

[0044] In operation S230, a target video frame is obtained based on the motion information of the virtual object and the background video frame.

[0045] In one example, the motion information of the virtual object matches the motion information of the target object.

[0046] In one example, the motion information of a real object such as a person or an animal can be collected to obtain an original video. The original video is deframed offline to obtain the original video frame. The motion of the target object in the original video is simulated to obtain the motion information of the virtual object that is consistent with the motion information of the target object.

[0047] In one example, the background of the area where the target object is located in the original video frame is repaired, and the obtained background video frame is a video frame after the area where the target object is located is filled with background information. For example, the foreground information containing the target object is removed, and the background information blocked by the target object is supplemented while retaining the visible background information to complete the background repair.

[0048] According to the embodiments of the present disclosure, based on the action information of the virtual object, the virtual object is integrated into the background video frame to obtain the target video frame, thereby replacing the real target object with the virtual object, combining the virtual object with the real scene, improving the novelty of the video content, and thus improving the user experience.

[0049] In a related example, the virtual object may replace the target object in the original video frame to obtain a target video frame in which the virtual object is combined with the real background scene.

[0050] In a related example, replacing the target object in the original video frame with the virtual object may include: fusing the virtual object into the original video frame, and using the virtual object to cover the target object to obtain the target video frame.

[0051] Compared with the method of directly fusing virtual objects into the original video frames, the video generation method provided by the embodiments of the present disclosure can improve the fusion effect and avoid the problems of cluttered background information and high background noise caused by incomplete occlusion of target objects.

[0052] According to the embodiments of the present disclosure, Figure 2The operation S220 shown, performing background repair on the area where the target object is located in the original video frame to obtain a background video frame, may include: using a mask image of the target object to perform background repair on the area where the target object is located in the original video frame to obtain a background video frame.

[0053] In one example, the mask image is used to indicate the pixel position of the target object in the original video frame. For example, the mask image can be a black and white image. The area where the target object is located can be set to black, and the background area can be set to white. However, it is not limited to this. As long as it can indicate the area where the target object is located and the background area, it can be used.

[0054] In one example, the pixel position may include a pixel-level position. However, it is not limited thereto. The mask image may also be used to indicate the image position of the target object in the original video frame. The image position may refer to the boundary point position of the bounding box. Compared with the image position, the pixel position has a finer granularity, down to the pixel level, thereby improving the accuracy of background restoration.

[0055] In one example, using the mask image of the target object to perform background repair on the area where the target object is located in the original video frame may include: restoring background information of the background area blocked by the target object to obtain a background video frame with complete background information.

[0056] According to the embodiments of the present disclosure, the target object can be effectively distinguished from the background object by using the mask map of the target object, thereby improving the restoration effect of the background information blocked by the target object. In addition, the mask map is refined to the pixel level, which improves the degree of refinement and thus improves the accuracy of background restoration.

[0057] According to an embodiment of the present disclosure, when executing Figure 2 In operation S220 shown, before performing background repair on the area where the target object is located in the original video frame to obtain the background video frame, the video generation method may further include: generating a mask image.

[0058] According to an embodiment of the present disclosure, generating a mask map may include: performing instance segmentation on an original video frame to obtain a mask map of a target object.

[0059] According to the embodiments of the present disclosure, instance segmentation is performed on the original video frame, so that the mask map of the target object can be refined to the pixel level, the target object and the background object can be finely classified, and the accuracy of the obtained mask map can be improved.

[0060] According to another embodiment of the present disclosure, generating a mask map may further include: performing instance segmentation on the original video frame based on the initial detection result to obtain a mask map of the target object.

[0061] According to an embodiment of the present disclosure, the initial detection result is obtained by performing target detection on the original video frame. For example, target detection is performed on the original video frame to obtain the initial detection result.

[0062] In one example, a YOLO (You Only Look Once, an object recognition and positioning algorithm based on a deep neural network) model can be used to detect the target on the original video frame to obtain an initial detection result. The initial detection result may include: the position of the detection box, the size of the detection box, and the target object category. The YOLO model may include a YOLO model or an improved network of the YOLO model such as YOLOv8.

[0063] According to the embodiments of the present disclosure, the target object can be quickly located by utilizing the initial detection results, thereby improving recognition efficiency.

[0064] In one example, based on the initial detection result, the original video frame is segmented to obtain a mask of the target object, which may include: based on the position of the detection frame, intercepting the target object image from the original video frame. The target object image is segmented by SAM (segment anything) calculation to obtain a mask of the target object. The SAM algorithm can perform fine segmentation on the target object according to the edge information and texture features of the target object, obtain the precise outline of the target object, and finally obtain a mask of the target object.

[0065] According to the embodiments of the present disclosure, dual detection of target detection and instance segmentation is utilized to improve the recognition accuracy of the mask map of the target object while improving the operational stability, thereby avoiding the problem of large amount of computational data caused by directly performing instance segmentation.

[0066] According to an embodiment of the present disclosure, performing instance segmentation on the original video frame to obtain a mask map of the target object may also include: performing instance segmentation on the original video frame to obtain an initial mask map of the target object. Filtering the initial mask map to obtain a mask map of the target object.

[0067] In one example, a mask image of a target object obtained using an instance segmentation algorithm may have defects, such as artifacts.

[0068] In one example, the mask image of the target object obtained by instance segmentation may be used as the initial mask image, and the initial mask image may be filtered to remove artifacts at the boundary, thereby obtaining the mask image of the target object with the boundary artifacts removed.

[0069] In one example, a filter frame of a predetermined size may be set to perform an outward expansion filter on the boundary of the initial mask image, thereby eliminating boundary artifacts.

[0070] According to an embodiment of the present disclosure, after performing instance segmentation on the original video frame to obtain a mask map of the target object, the video generation method may further include: determining whether the mask map of the target object is abnormal. When it is determined that the mask map of the target object is abnormal, the abnormal mask map is corrected using an adjacent mask map to obtain a corrected mask map.

[0071] In one example, the adjacent mask map corresponds to adjacent video frames of the original video frame.

[0072] In one example, the modified mask image is used to perform background repair on the area where the target object is located in the original video frame to obtain a background video frame.

[0073] In one example, determining whether the mask image of the target object is abnormal may include: performing abnormal image recognition on the mask image sequence to obtain an abnormal mask image recognition result. Based on the abnormal mask image recognition result, determining whether the mask image of the target object is abnormal.

[0074] In one example, the original video may be deframed to obtain a plurality of original video frames arranged in time sequence, such as an original video frame sequence, and each original video frame in the original video frame sequence may be instance segmented to obtain a mask image sequence corresponding to the original video frame sequence.

[0075] In one example, identifying abnormal images for a mask image sequence may include: comparing the pixel positions of the target objects of each of the multiple mask images. Determining a difference value based on the pixel position and the mean of the pixel position. Determining a difference value greater than a predetermined threshold as an abnormal mask image. However, it is not limited to this. It may also be possible to determine a difference value based on the pixel region area of ​​the target object and the mean of the pixel region area. Determining a difference value greater than a predetermined threshold as an abnormal mask image. As long as the method of determining the abnormal mask image identification result is based on the mask image sequence of the original video frame sequence, it is acceptable.

[0076] In one example, the abnormal mask map may be corrected using adjacent mask maps corresponding to adjacent video frames to obtain a corrected mask map.

[0077] Compared with the method of deleting the abnormal mask image, correcting the abnormal mask image can avoid the problem of frame skipping and make the subsequently generated target video coherent.

[0078] Compared with the method of directly using the abnormal mask map for subsequent operations, correcting the abnormal mask map can avoid the problem of incomplete background area repair, so that the background information of the subsequently generated target video is complete.

[0079] In one example, using the adjacent mask map to correct the abnormal mask map to obtain the corrected mask map may include: performing pixel comparison on the abnormal mask map based on the adjacent mask map to obtain cross pixels. Performing pixel conversion on the cross pixels to obtain the corrected mask map. The cross pixels may refer to pixels obtained after taking the intersection or the union. Whether to take the intersection or the union may be determined according to actual conditions.

[0080] Figure 3 A schematic diagram of a modified exception mask map according to an embodiment of the present disclosure is schematically shown.

[0081] like Figure 3 As shown, batch instance segmentation is performed on multiple original video frames to obtain a first mask image 310, a second mask image 320 and a third mask image 330 of a target object 340. Abnormal image recognition is performed on the first mask image 310, the second mask image 320 and the third mask image 330 to obtain an abnormal mask image recognition result. For example, the pixel position or pixel area area of ​​each target object of the mask image sequence is compared. The pixel position or pixel area area abnormal mask image is determined as an abnormal mask image. It is determined that the difference between the pixel area area of ​​the target object of the second mask image 320 and the pixel area area of ​​the target object of the first mask image 310 is greater than a predetermined threshold, and the difference between the pixel area area of ​​the target object of the second mask image 320 and the pixel area area of ​​the target object of the third mask image 330 is greater than a predetermined threshold. Determine that the second mask image 320 is an abnormal mask image.

[0082] like Figure 3 As shown, the second mask image 320 is corrected using the first mask image 310 and the third mask image 330 to obtain a corrected mask image 320'.

[0083] According to an embodiment of the present disclosure, using adjacent mask images to correct an abnormal mask image may include: using the average of the pixel area areas of multiple adjacent mask images as the pixel area area of ​​the abnormal mask image, or using the average of the pixel positions of multiple adjacent mask images as the pixel position of the abnormal mask image.

[0084] According to the embodiments of the present disclosure, by utilizing the continuity principle between multiple original video frames in the original video frame sequence, an abnormal mask map can be determined and the abnormal mask map can be corrected using adjacent mask maps, thereby ensuring processing accuracy and improving processing efficiency.

[0085] Figure 4 The flowchart of generating a background video frame according to an embodiment of the present disclosure is schematically shown.

[0086] like Figure 4 As shown, generating a background video frame may include: operation S410, operation S420, operation S431 and operation S432.

[0087] In operation S410, a background region is dynamically identified based on a plurality of original video frames to obtain a plurality of identification results, wherein the identification results indicate whether the background region in the original video frame includes a dynamic background object.

[0088] In operation S420, based on the recognition result, it is determined whether the background object is moving. In the case where the recognition result indicates that the background area in the original video frame includes a dynamic background object, such as the background object is moving, operation S431 is performed. In the case where it is determined that the background area in the original video frame includes a static background object, such as the background object is not moving, operation S432 is performed.

[0089] In operation S431, the original video frame is repaired by using a background repair method to obtain a background video frame.

[0090] In operation S432, the original video frame is repaired by using a global optimization method to obtain a background video frame.

[0091] In one example, based on multiple original video frames, dynamic recognition of the background area is performed to obtain the recognition result, which may include: using a frame difference method and / or an optical flow method to judge the multiple original video frames to determine whether the background object in the background area is moving, and if there is no movement, it means that the background object is motionless and static. If the image position of the background object changes with time, it means that the background object is moving.

[0092] In one example, operations S410 and S420 may not be performed, for example, the camera motion is not determined, and the original video frame is directly repaired by using a background repair method or a global optimization method.

[0093] Compared with the method of not performing operations S410 and S420, by using operations S410 and S420, a target restoration method corresponding to repairing the original video frame, such as a background restoration method or a global optimization method, can be determined. The original video frame can be repaired by using the target restoration method, which is targeted and flexible, and improves the scope of application while improving the restoration accuracy, naturalness and consistency.

[0094] According to an embodiment of the present disclosure, repairing the original video frame in a global optimization manner to obtain a background video frame may include: obtaining the background video frame based on a plurality of original video frames and a plurality of mask images of target objects.

[0095] In one example, since the background object in the background area is static, the background information of the background areas of the multiple original video frames is the same. Based on the mask images of the multiple target objects, the one with the smallest blocked background area can be determined from the multiple original video frames as the background video frame.

[0096] In another example, background regions may be extracted from multiple original video frames based on mask images of multiple target objects, and background video frames may be obtained using background information of each of the multiple background regions.

[0097] For example, based on multiple original video frames and multiple mask images of target objects, multiple intermediate video frames are obtained. The intermediate video frames include images without the target objects. Based on image information of background areas of the multiple intermediate video frames, the target intermediate video frames are background-completed to obtain background video frames.

[0098] In one example, the target intermediate video frame is one of the multiple intermediate video frames.

[0099] According to the embodiments of the present disclosure, since the target object is moving and the background object is static, the target object is located in different image positions in multiple original video frames, and the blocked background areas are different. The background information of the unblocked background areas in the multiple original video frames is used to complete the background of the blocked background areas in the target intermediate video frame, so as to obtain a background video frame with complete background information. The background video frame can be obtained without using large amounts of data for calculation, thereby improving processing efficiency. In addition, the pixel information of each of the multiple original video frames is consistent, and the background is completed using high-pixel image information to ensure image clarity and resolution, so that the restoration effect is good.

[0100] Figure 5 A schematic diagram of generating a background video frame according to an embodiment of the present disclosure is schematically shown.

[0101] When it is determined that the background object is a static background object, the target object may be removed from the original video frame based on the mask image of the target object to obtain an intermediate video frame.

[0102] like Figure 5 As shown, in the middle video frame 510, the target object blocks the background information such as stars and crosses in the right part of the background area, while the background information in the left part of the background area is complete. In the target middle video frame 520, the target object blocks the background information such as stars and crosses in the left part of the background area, while the background information in the right part of the background area is complete. The image information of the background area with complete background information in the middle video frame 510 can be used to fill the target middle video frame 520, and the background of the target middle video frame 520 is completed to obtain the background video frame 530.

[0103] According to the embodiments of the present disclosure, a global optimization method is used to reduce the demand for hardware devices such as video memory, and the processing speed is fast and the restoration effect is good without being affected by factors such as resolution and video length of the original video.

[0104] According to an embodiment of the present disclosure, repairing the original video frame by using a background repair method to obtain a background video frame may include: fusing the mask image and the original video frame to obtain a fused image. Extracting local features and global features from the fused image to obtain local features and global features. Cross-fusing the local features and the global features to obtain the background video frame.

[0105] In one example, fusing the mask image with the original video frame to obtain a fused image may include: superimposing channel information of the mask image with the original video frame to obtain a fused image. However, the present invention is not limited thereto, and the mask image may be inverted and pixel-multiplied with the original video frame to obtain an initial fused image. The initial fused image and the mask image are superimposed on channel information to obtain a fused image.

[0106] In one example, two branches may be used to extract local features and global features from the fused image, respectively, to obtain local features and global features. Each branch may be processed by Fast Fourier Convolutions (FFCs).

[0107] According to the embodiments of the present disclosure, by combining the global feature extraction and the local feature extraction, the global information and the local information of the fused image can be extracted, thereby improving the restoration effect of the background video frame.

[0108] According to an embodiment of the present disclosure, the mask image and the original video frame can be processed by an image inpainting model to obtain a background video frame. The image inpainting model can include an initial fusion module, a global feature extraction module, a local feature extraction module and a cross fusion module.

[0109] Figure 6 The structural diagram of the image inpainting model according to the embodiment of the present disclosure is schematically shown.

[0110] like Figure 6 As shown, the original video frame 610 and the mask image 620 are input into the initial fusion module M610, and the mask image 620 and the original video frame 610 are fused by the initial fusion module M610 to obtain a fused image 630. The fused image 630 is respectively input into the global feature extraction module M620 and the local feature extraction module M630, and the global feature extraction module M620 is used to extract the global features of the fused image 630 to obtain the global features 640. The local feature extraction module M630 is used to extract the local features of the fused image 630 to obtain the local features 650. The global features 640 and the local features 650 are input into the cross fusion module M640, and the local features 650 and the global features 640 are cross fused by the cross fusion module M640 to obtain the background video frame 660.

[0111] According to the embodiments of the present disclosure, the background video frame is obtained by using an image repair model, which has low requirements on hardware devices such as video memory, high processing efficiency and is independent of the video length of the original video, has a wide range of applications and good repair effects.

[0112] According to the embodiments of the present disclosure, Figure 2 The operation S230 shown, performing motion capture on the target object in the original video frame to generate motion information of the virtual object, may include: performing motion capture on the target object in the original video frame to generate posture information of the virtual object, obtaining a posture information sequence of the original video frame sequence, and obtaining a motion information sequence of the virtual object based on the posture information sequence.

[0113] In one example, the original video frame sequence includes a plurality of original video frames arranged in time sequence, and the action information sequence includes action information of a virtual object matching the original video frames.

[0114] In one example, a single-frame motion capture can be performed on the target object in each original video frame to generate the posture information of the virtual object. Based on the posture information of the virtual objects in multiple original video frames in the original video frame sequence, a posture information sequence corresponding to the original video frame sequence is obtained.

[0115] In one example, the posture information sequence can be used as the action information sequence of the virtual object. However, it is not limited to this. The posture information sequence can also be smoothed to obtain the action information sequence of the virtual object. Smoothing can refer to smooth transition processing of the posture information sequence in the timing process.

[0116] According to the embodiments of the present disclosure, based on the posture information sequence, the action information sequence of the target object is obtained, and the action can be reconstructed by utilizing the temporal correlation between adjacent original video frames and the action transformation relationship between multiple original video frames, thereby improving the reconstruction effect of the action information sequence of the virtual object and improving the continuity and smoothness of the temporal action.

[0117] According to an embodiment of the present disclosure, the target object in the original video frame is motion captured to generate the posture information of the virtual object, including: performing key point recognition on the target object in the original video frame to obtain key point parameter information. Based on the key point parameter information, three-dimensional posture parameters are obtained. Two-dimensional projection is performed on the three-dimensional posture parameters to obtain posture information.

[0118] In one example, key point parameter information, such as key point parameters of a human body three-dimensional model (Skinned Multi-Person Linear Model, SMPL), may include limb key point parameter information and hand key point parameter information. Limb key point parameter information is obtained by limb key point recognition. Hand key point parameter information is obtained by hand key point recognition.

[0119] For example, the limb key point recognition model can be used to recognize limb key points on the original video frame to obtain limb key point parameter information, such as β and θ. β represents 10 parameters of human body height, weight, head-to-body ratio, etc. θ represents 75 parameters of the overall motion posture of the human body and the relative angles of 24 joints. The hand key point recognition model can be used to recognize hand key points on the original video frame to obtain hand key point parameter information, such as key point parameters such as hand posture and shape. The model structure of the limb key point recognition model and the hand key point recognition model is not limited, as long as it is a model that can realize key point recognition.

[0120] In one example, three-dimensional posture parameters including joint information of limbs and hands can be obtained based on key point parameter information. The three-dimensional posture parameters are projected two-dimensionally through camera parameters to obtain posture information.

[0121] According to the embodiments of the present disclosure, limb key point recognition and hand key point recognition are performed on the target object respectively, and the key point recognition granularity is fine, thereby improving the recognition accuracy of the posture information.

[0122] According to an embodiment of the present disclosure, obtaining an action information sequence of a virtual object based on a posture information sequence may include: performing spatiotemporal dual-stream processing on the posture information sequence to obtain an action information sequence of the virtual object.

[0123] In one example, the spatiotemporal dual-stream model can be used to perform spatiotemporal dual-stream processing on the posture information sequence to obtain the action information sequence of the virtual object. The spatiotemporal dual-stream model can include two deep convolutional networks, which independently use static image frames and dense optical flows between frames to perform action information recognition, and finally fuse the results to obtain the action information sequence. However, it is not limited to this, as long as the network structure can extract action coherence features from the posture information sequence containing the temporal relationship and obtain a coherent action information sequence of the virtual object.

[0124] According to an embodiment of the present disclosure, a spatiotemporal dual-stream processing is performed on a posture information sequence, and the posture information obtained based on a single frame is combined with the timing features, thereby making the obtained action information sequences of the virtual object coherent and smooth with each other, thereby improving the realism of the virtual object simulation.

[0125] According to an embodiment of the present disclosure, obtaining a target video frame based on the action information of the virtual object and the background video frame may include: generating a 3D reconstruction model of the virtual object based on the action information of the virtual object; fusing the 3D reconstruction model with the background video based on the image position of the target object in the original video frame to generate the target video frame.

[0126] In one example, a preset reference 3D model and action information may be combined to generate a 3D reconstruction model of a virtual object. However, this is not limited to this. A skin may also be designed based on the action information of the virtual object to obtain a 3D reconstruction model of the virtual object. As long as the 3D reconstruction model matches the action information of the virtual object, it will be fine.

[0127] In one example, based on the image position of the target object in the original video frame, for example, the trajectory coordinates in the motion trajectory, the three-dimensional reconstructed model and the background video are fused to generate a target video frame.

[0128] In one example, the synthesized target video frame may also be rendered and post-processed to achieve a final visual effect.

[0129] In one example, multiple target video frames may be combined into a target video, and the target video may be fed back to the user.

[0130] According to the embodiments of the present disclosure, by fusing the three-dimensional reconstructed model and the background video, the virtual object in the generated target video frame can be made to fit the target object not only in terms of movement, but also in terms of body shape such as height and fatness, thereby improving the authenticity of the target video frame.

[0131] Figure 7 The flowchart of a video generating method according to another embodiment of the present disclosure is schematically shown.

[0132] like Figure 7 As shown, the video generation method may include: performing instance segmentation on the original video frame 710 to obtain a mask image 720 of the target object. Based on the mask image 720 of the target object, performing background restoration on the original video frame 710 to obtain a background video frame 730. Performing motion capture on the original video frame 710 to obtain posture information of the virtual object. Based on the posture information, obtaining a three-dimensional reconstruction model of the virtual object, such as a virtual object model 740. Synthesizing the virtual object model 740 and the background video frame 730 to obtain a target video frame 750.

[0133] Figure 8 The figure schematically shows a block diagram of a video generating device according to an embodiment of the present disclosure.

[0134] like Figure 8 As shown, the video processing device includes: a capturing module 810 , a repairing module 820 and a synthesizing module 830 .

[0135] The capture module 810 is used to perform motion capture on the target object in the original video frame to generate motion information of the virtual object. The motion information of the virtual object matches the motion information of the target object.

[0136] The restoration module 820 is used to perform background restoration on the area where the target object is located in the original video frame to obtain a background video frame.

[0137] The synthesis module 830 is used to obtain a target video frame based on the action information of the virtual object and the background video frame.

[0138] According to an embodiment of the present disclosure, the repair module includes: a repair submodule.

[0139] The repair submodule is used to perform background repair on the area where the target object is located in the original video frame using the mask image of the target object to obtain a background video frame. The mask image is used to indicate the pixel position of the target object in the original video frame.

[0140] According to an embodiment of the present disclosure, the video generating device further includes: a segmentation module.

[0141] The segmentation module is used to perform instance segmentation on the original video frame to obtain the mask map of the target object.

[0142] According to an embodiment of the present disclosure, the segmentation module includes: a segmentation submodule.

[0143] The segmentation submodule is used to perform instance segmentation on the original video frame based on the initial detection result to obtain a mask map of the target object. The initial detection result is obtained by performing target detection on the original video frame.

[0144] According to an embodiment of the present disclosure, the segmentation submodule includes: a segmentation unit and a filtering unit.

[0145] The segmentation unit is used to perform instance segmentation on the original video frame to obtain an initial mask map of the target object.

[0146] The filtering unit is used to filter the initial mask image to obtain a mask image of the target object.

[0147] According to an embodiment of the present disclosure, the video generating device further includes: a mask correction module.

[0148] The mask correction module is used to correct the abnormal mask map using the adjacent mask map when it is determined that the mask map of the target object has an abnormality, so as to obtain a corrected mask map. The corrected mask map is used to perform background repair on the area where the target object is located in the original video frame. The adjacent mask map corresponds to the adjacent video frame of the original video frame.

[0149] According to an embodiment of the present disclosure, the repair submodule includes: a static repair unit.

[0150] The static restoration unit is used to obtain a background video frame based on a plurality of original video frames and a plurality of mask images of target objects when it is determined that the background area in the original video frame includes a static background object.

[0151] According to an embodiment of the present disclosure, the static repair unit includes: a first static repair subunit and a second static repair subunit.

[0152] The first static restoration subunit is used to obtain a plurality of intermediate video frames based on a plurality of original video frames and a plurality of mask images of target objects. The intermediate video frames include images without the target objects.

[0153] The second static restoration subunit is used to perform background completion on a target intermediate video frame based on image information of background regions of each of the multiple intermediate video frames to obtain a background video frame. The target intermediate video frame is one of the multiple intermediate video frames.

[0154] According to an embodiment of the present disclosure, the repair submodule includes: an initial fusion unit, a feature extraction unit and a cross fusion unit.

[0155] The initial fusion unit is used to fuse the mask image and the original video frame to obtain a fused image when it is determined that the background area in the original video frame includes a dynamic background object.

[0156] The feature extraction unit is used to extract local features and global features from the fused image to obtain local features and global features.

[0157] The cross fusion unit is used to cross-fuse local features and global features to obtain a background video frame.

[0158] According to an embodiment of the present disclosure, the video generating device further includes: a dynamic recognition module.

[0159] The dynamic recognition module is used to dynamically recognize the background area based on multiple original video frames to obtain a recognition result. The recognition result indicates whether the background area in the original video frame includes a dynamic background object.

[0160] According to an embodiment of the present disclosure, the capture module includes: a posture capture submodule and a motion capture submodule.

[0161] The gesture capture submodule is used to capture the motion of the target object in the original video frame, generate the gesture information of the virtual object, and obtain the gesture information sequence of the original video frame sequence. The original video frame sequence includes multiple original video frames arranged in time sequence.

[0162] The motion capture submodule is used to obtain a motion information sequence of the virtual object based on the posture information sequence, wherein the motion information sequence includes the motion information of the virtual object that matches the original video frame.

[0163] According to an embodiment of the present disclosure, the gesture capture submodule includes: a key point recognition unit, a conversion unit and a projection unit.

[0164] The key point recognition unit is used to recognize the key points of the target object in the original video frame to obtain key point parameter information. The key point parameter information includes limb key point parameter information and hand key point parameter information. The limb key point parameter information is obtained through limb key point recognition, and the hand key point parameter information is obtained through hand key point recognition.

[0165] The conversion unit is used to obtain three-dimensional posture parameters based on key point parameter information.

[0166] The projection unit is used to perform two-dimensional projection on the three-dimensional posture parameters to obtain posture information.

[0167] According to an embodiment of the present disclosure, the motion capture submodule includes: a motion capture unit.

[0168] The motion capture unit is used to perform spatiotemporal dual-stream processing on the posture information sequence to obtain the motion information sequence of the virtual object.

[0169] According to an embodiment of the present disclosure, the synthesis module includes: a reconstruction submodule and a synthesis submodule.

[0170] The reconstruction submodule is used to generate a three-dimensional reconstruction model of the virtual object based on the action information of the virtual object.

[0171] The synthesis submodule is used to fuse the three-dimensional reconstructed model and the background video based on the image position of the target object in the original video frame to generate the target video.

[0172] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0173] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a method as in the embodiment of the present disclosure.

[0174] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute a method as described in an embodiment of the present disclosure.

[0175] According to an embodiment of the present disclosure, a computer program product includes a computer program, and when the computer program is executed by a processor, the method according to the embodiment of the present disclosure is implemented.

[0176] Fig. 9 A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0177] like Fig. 9 As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0178] A number of components in the device 900 are connected to an input / output (I / O) interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0179] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as a video generation method. For example, in some embodiments, the video generation method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by the computing unit 901, one or more steps of the video generation method described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to perform the video generation method in any other appropriate manner (e.g., by means of firmware).

[0180] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0181] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0182] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0183] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0184] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0185] A computer system may include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises through computer programs running on respective computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server combined with a blockchain.

[0186] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0187] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A video processing method, comprising: Performing motion capture on a target object in an original video frame to generate posture information of a virtual object, and obtaining a posture information sequence of an original video frame sequence, wherein the original video frame sequence includes a plurality of the original video frames arranged in time sequence; Performing spatiotemporal dual-stream processing on the posture information sequence to obtain an action information sequence of the virtual object, wherein the action information sequence includes action information of the virtual object that matches the original video frame, and the action information of the virtual object matches the action information of the target object; Performing background restoration on the area where the target object is located in the original video frame to obtain a background video frame, wherein the background restoration includes restoring background information blocked by the target object in the original video frame; and A target video frame is obtained based on the action information of the virtual object and the background video frame.

2. The method according to claim 1, wherein: The performing background repair on the area where the target object is located in the original video frame to obtain a background video frame includes: The target object's mask image is used to perform background restoration on the area where the target object is located in the original video frame to obtain the background video frame, wherein the mask image is used to indicate the pixel position of the target object in the original video frame.

3. The method according to claim 2, further comprising: Perform instance segmentation on the original video frame to obtain a mask map of the target object.

4. The method according to claim 3, wherein: The performing instance segmentation on the original video frame to obtain a mask map of the target object includes: Based on the initial detection result, instance segmentation is performed on the original video frame to obtain a mask map of the target object, wherein the initial detection result is obtained by performing target detection on the original video frame.

5. The method according to claim 3, wherein: The performing instance segmentation on the original video frame to obtain a mask map of the target object includes: Performing instance segmentation on the original video frame to obtain an initial mask map of the target object; and The initial mask image is filtered to obtain a mask image of the target object.

6. The method according to any one of claims 2 to 5, further comprising: When it is determined that there is an abnormality in the mask map of the target object, the mask map with the abnormality is corrected using an adjacent mask map to obtain a corrected mask map, wherein the adjacent mask map corresponds to an adjacent video frame of the original video frame, and the corrected mask map is used to perform background repair on the area where the target object is located in the original video frame.

7. The method according to any one of claims 2 to 5, wherein: The method of using the mask image of the target object to perform background repair on the area where the target object is located in the original video frame to obtain the background video frame includes: In the case where it is determined that the background area in the original video frame includes a static background object, the background video frame is obtained based on a plurality of the original video frames and a plurality of mask images of the target objects.

8. The method according to claim 7, wherein: The obtaining of the background video frame based on the plurality of original video frames and the plurality of mask images of the target objects comprises: Based on the plurality of original video frames and the plurality of mask images of the target objects, a plurality of intermediate video frames are obtained, wherein the intermediate video frames include images without the target objects; and Based on the image information of the background areas of the respective multiple intermediate video frames, the background of the target intermediate video frame is completed to obtain the background video frame, wherein the target intermediate video frame is one of the multiple intermediate video frames.

9. The method according to any one of claims 2 to 5, wherein: The method of using the mask image of the target object to perform background repair on the area where the target object is located in the original video frame to obtain the background video frame includes: When it is determined that the background area in the original video frame includes a dynamic background object, fusing the mask image with the original video frame to obtain a fused image; Performing local feature extraction and global feature extraction on the fused image respectively to obtain local features and global features; and The local features and the global features are cross-fused to obtain the background video frame.

10. The method according to claim 7, further comprising: Based on the multiple original video frames, the background area is dynamically identified to obtain an identification result, wherein the identification result represents whether the background area in the original video frame includes a dynamic background object.

11. The method according to claim 1, wherein: The step of performing motion capture on the target object in the original video frame to generate posture information of the virtual object includes: Performing key point recognition on the target object in the original video frame to obtain key point parameter information, wherein the key point parameter information includes limb key point parameter information and hand key point parameter information, the limb key point parameter information is obtained through limb key point recognition, and the hand key point parameter information is obtained through hand key point recognition; Based on the key point parameter information, obtain three-dimensional posture parameters; and Performing a two-dimensional projection on the three-dimensional posture parameters to obtain the posture information.

12. The method according to any one of claims 1 to 5, wherein: The step of obtaining a target video based on the action information of the virtual object and the background video frame comprises: generating a three-dimensional reconstruction model of the virtual object based on the motion information of the virtual object; and Based on the image position of the target object in the original video frame, the three-dimensional reconstructed model and the background video are fused to generate the target video.

13. A video processing device, comprising: A capture module, used to perform motion capture on a target object in an original video frame to generate motion information of a virtual object, wherein the motion information of the virtual object matches the motion information of the target object; a repair module, configured to perform background repair on the area where the target object is located in the original video frame to obtain a background video frame, wherein the background repair includes restoring the background information blocked by the target object in the original video frame; and A synthesis module, used for obtaining a target video frame based on the action information of the virtual object and the background video frame; Wherein, the capture module includes: a gesture capture submodule, configured to perform motion capture on a target object in the original video frame, generate gesture information of the virtual object, and obtain a gesture information sequence of an original video frame sequence, wherein the original video frame sequence includes a plurality of the original video frames arranged in time sequence; and A motion capture submodule, configured to obtain a motion information sequence of the virtual object based on the posture information sequence, wherein the motion information sequence includes motion information of the virtual object that matches the original video frame; Wherein, the motion capture submodule includes: The motion capture unit is used to perform spatiotemporal dual-stream processing on the posture information sequence to obtain the motion information sequence of the virtual object.

14. The device according to claim 13, wherein: The repair module comprises: The repair submodule is used to use the mask map of the target object to perform background repair on the area where the target object is located in the original video frame to obtain the background video frame, wherein the mask map is used to indicate the pixel position of the target object in the original video frame.

15. The apparatus according to claim 14, further comprising: The segmentation module is used to perform instance segmentation on the original video frame to obtain a mask map of the target object.

16. The device according to claim 15, wherein: The segmentation module comprises: The segmentation submodule is used to perform instance segmentation on the original video frame based on the initial detection result to obtain a mask map of the target object, wherein the initial detection result is obtained by performing target detection on the original video frame.

17. The device according to claim 16, wherein: The segmentation submodule comprises: a segmentation unit, configured to perform instance segmentation on the original video frame to obtain an initial mask map of the target object; and A filtering unit is used to filter the initial mask image to obtain the mask image of the target object.

18. The device according to any one of claims 14 to 17, further comprising: A mask correction module is used to correct the mask map with the abnormality using an adjacent mask map when it is determined that the mask map of the target object has an abnormality, so as to obtain a corrected mask map, wherein the adjacent mask map corresponds to an adjacent video frame of the original video frame, and the corrected mask map is used to perform background repair on the area where the target object is located in the original video frame.

19. The device according to any one of claims 14 to 17, wherein: The repair submodule includes: The static restoration unit is used to obtain the background video frame based on a plurality of the original video frames and a plurality of mask images of the target objects when it is determined that the background area in the original video frame includes a static background object.

20. The device according to claim 19, wherein The static repair unit comprises: A first static restoration subunit is configured to obtain a plurality of intermediate video frames based on the plurality of original video frames and the plurality of mask images of the target objects, wherein the intermediate video frames include images without the target objects; and The second static restoration subunit is used to perform background completion on a target intermediate video frame based on image information of the background areas of each of the multiple intermediate video frames to obtain the background video frame, wherein the target intermediate video frame is one of the multiple intermediate video frames.

21. The device according to any one of claims 14 to 17, wherein: The repair submodule includes: an initial fusion unit, configured to fuse the mask image and the original video frame to obtain a fused image when it is determined that the background area in the original video frame includes a dynamic background object; a feature extraction unit, configured to extract local features and global features from the fused image to obtain local features and global features; and A cross-fusion unit is used to cross-fuse the local features and the global features to obtain the background video frame.

22. The apparatus of claim 19, further comprising: The dynamic recognition module is used to dynamically recognize the background area based on the multiple original video frames to obtain a recognition result, wherein the recognition result represents whether the background area in the original video frame includes a dynamic background object.

23. The device according to claim 13, wherein: The gesture capture submodule comprises: A key point recognition unit, used for performing key point recognition on the target object in the original video frame to obtain key point parameter information, wherein the key point parameter information includes limb key point parameter information and hand key point parameter information, the limb key point parameter information is obtained through limb key point recognition, and the hand key point parameter information is obtained through hand key point recognition; A conversion unit, configured to obtain three-dimensional posture parameters based on the key point parameter information; and The projection unit is used to perform two-dimensional projection on the three-dimensional posture parameters to obtain the posture information.

24. The device according to any one of claims 13 to 17, wherein: The synthesis module comprises: a reconstruction submodule, configured to generate a three-dimensional reconstruction model of the virtual object based on the motion information of the virtual object; and The synthesis submodule is used to fuse the three-dimensional reconstructed model and the background video based on the image position of the target object in the original video frame to generate the target video.

25. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 12.

26. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 12.

27. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 12.

Images