Video generation method, computer terminal and computer readable storage medium
By identifying and generating smooth timing local focus, the problem of inter-frame inconsistency in video virtual trial-on is solved, the quality and detail retention of video presentation are improved, and the better video virtual trial-on effect is achieved.
Patent Information
- Application Number
- CN202410985866.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-22
- Publication Date
- 2024-11-22
AI Technical Summary
In the existing video virtual trial-on technology, inter-frame inconsistency leads to poor video presentation effects and lacks effective solutions.
By obtaining the original video of the biological object and the image information of the items to be displayed, identify the target item and generate a smooth local focus with timing, use the smooth local focus with timing, and input the video to generate the target video to ensure that the target item is in Stable close-ups and details in the video are retained.
Improve the video effect of virtual trial on video, provide more accurate placement of target items and display details, and improve the quality of video presentation.
Smart Images

Figure CN119011749A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the fields of artificial intelligence and video generation, and in particular, to a video generation method, a computer terminal, and a computer-readable storage medium. Background Art
[0002] Video virtual try-on aims to put the specified clothing or accessories on the target person in the video sequence, providing consumers with an interactive experience. It has important application value in many application scenarios such as e-commerce, clothing customization, clothing design and production.
[0003] Compared with image virtual try-on, video virtual try-on adds the time dimension and can provide a comprehensive presentation of try-on from different perspectives. At present, for video virtual try-on, the image virtual try-on method is usually used to process the video frame by frame, but the generated video has significant inconsistency between frames, resulting in poor presentation of video virtual try-on.
[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention
[0005] The embodiments of the present application provide a video generation method, a computer terminal, and a computer-readable storage medium to at least solve the technical problem of poor video effects presented by video virtual try-on.
[0006] According to one aspect of an embodiment of the present application, a video generation method is provided, comprising: obtaining an original video of a biological object and image information of an object to be displayed, wherein the object to be displayed is an object to be displayed on the biological object; identifying a target object in the original video and generating a temporally smooth local focus, wherein the temporally smooth local focus is used to characterize position information of the target object in different video frames of the original video, and the target object is an object currently loaded on the biological object as a carrier; amplifying the original video using the temporally smooth local focus to obtain an amplified video; inputting the amplified video into a video generation model, and guiding the video generation model to generate a target video based on the temporally smooth local focus and image information, wherein the object to be displayed is displayed on the biological object in the target video.
[0007] According to another aspect of the embodiment of the present application, a video generation method is also provided, including: in response to an image input instruction acting on an operation interface, displaying an original video of a biological object and image information of an object to be displayed on the operation interface, wherein the object to be displayed is an object to be displayed on the biological object; in response to a generation instruction acting on the operation interface, displaying a target video on the operation interface, wherein the object to be displayed is displayed on the biological object in the target video, and the target video is a video generated based on the original video, the image information and time-smoothed local focus, the time-smoothed local focus is generated by identifying the target object in the original video, and the time-smoothed local focus is used to characterize the position information of the target object in different video frames of the original video, and the target object is an object that has been displayed on the biological object.
[0008] According to another aspect of an embodiment of the present application, a video generation method is also provided, including: obtaining an original video of a user and image information of a first article of clothing, wherein the first article of clothing is the article of clothing to be worn by the user; identifying a second article of clothing in the original video, and generating a temporally smooth local focus, wherein the temporally smooth local focus is used to characterize position information of the second article of clothing in different video frames of the original video, and the second article of clothing is the article of clothing already worn by the user; generating a target video based on the original video, the image information and the temporally smooth local focus, wherein the user in the target video wears the first article of clothing.
[0009] According to another aspect of an embodiment of the present application, a video generation method is also provided, including: displaying an original video of a biological object and image information of an object to be displayed on a presentation screen of a virtual reality VR device or an augmented reality AR device, wherein the object to be displayed is an object to be displayed on the biological object; identifying the target object in the original video and generating a temporally smooth local focus, wherein the temporally smooth local focus is used to characterize position information of the target object in different video frames of the original video, and the target object is an object that has been displayed on the biological object; generating a target video based on the original video, the image information and the temporally smooth local focus, wherein the object to be displayed is displayed on the biological object in the target video; and driving the VR device or the AR device to display the target video.
[0010] According to another aspect of an embodiment of the present application, a video generation method is also provided, including: obtaining an original video of a biological object and image information of an object to be displayed by calling a first interface, wherein the first interface includes a first parameter, and the parameter value of the first parameter includes the original video and the image information, and the object to be displayed is an object to be displayed on the biological object; identifying the target object in the original video and generating a temporally smooth local focus, wherein the temporally smooth local focus is used to characterize the position information of the target object in different video frames of the original video, and the target object is an object that has been displayed on the biological object; generating a target video based on the original video, the image information and the temporally smooth local focus, wherein the object to be displayed is displayed on the biological object in the target video; outputting the target video by calling a second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter includes the target video.
[0011] According to another aspect of the embodiments of the present application, a computer terminal is further provided, including: a memory storing an executable program; and a processor for running the program, wherein the method in each embodiment of the present application is executed when the program is running.
[0012] According to another aspect of an embodiment of the present application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute the methods in each embodiment of the present application.
[0013] According to another aspect of the embodiments of the present application, a computer program product is also provided, including a computer program, and when the computer program is executed by a processor, the method in each embodiment of the present application is implemented.
[0014] According to another aspect of an embodiment of the present application, a computer program product is also provided, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method in each embodiment of the present application is implemented.
[0015] According to another aspect of the embodiments of the present application, a computer program is also provided, and when the computer program is executed by a processor, the methods in the various embodiments of the present application are implemented.
[0016] In an embodiment of the present application, an original video of a biological object and image information of an object to be displayed are obtained; the target object in the original video is identified to generate a temporally smooth local focus; the original video is amplified using the temporally smooth local focus to obtain an amplified video; the amplified video is input into a video generation model, and the video generation model is guided to generate a target video based on the temporally smooth local focus and image information. It is easy to notice that a stable close-up shot of the area where the target object is located in the original video is extracted by temporally smooth local focus, so as to achieve the purpose of accurately locating the target object, and the original video is amplified using temporally smooth local focus, thereby amplifying the area where the target object is located, better retaining the details of the target object, and providing a more accurate reference for the placement and display details of the object to be displayed, thereby improving the presentation effect of the target video generated based on the amplified video under the guidance of the temporally smooth local focus and image information, thereby solving the technical problem of poor video effect presented by the video virtual try-on.
[0017] It is easy to notice that the above general description and the following detailed description are only for the purpose of exemplifying and explaining the present application, and do not constitute a limitation of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0019] Figure 1 is a schematic diagram of a hardware environment of a virtual reality device according to a video generation method of an embodiment of the present application;
[0020] Figure 2 is a structural block diagram of a computing environment of a video generation method according to an embodiment of the present application;
[0021] Figure 3 is a flow chart of a video generation method according to an embodiment of the present application;
[0022] Figure 4 is a schematic diagram of an optional video generation model according to an embodiment of the present application;
[0023] Figure 5 is a schematic diagram of attention processing performed by an optional spatial attention module according to an embodiment of the present application;
[0024] Figure 6 is a schematic diagram of attention processing performed by an optional temporal attention module according to an embodiment of the present application;
[0025] Figure 7is a schematic diagram of an optional video generation method according to an embodiment of the present application;
[0026] Figure 8 is a flow chart of a video generation method according to an embodiment of the present application;
[0027] Fig. 9 is a flow chart of a video generation method according to an embodiment of the present application;
[0028] Fig.10 is a flow chart of a video generation method according to an embodiment of the present application;
[0029] Fig.11 is a flow chart of a video generation method according to an embodiment of the present application;
[0030] Fig.12 is a schematic diagram of a video generating device according to an embodiment of the present application;
[0031] Fig.13 is a schematic diagram of a video generating device according to an embodiment of the present application;
[0032] Fig.14 is a schematic diagram of a video generating device according to an embodiment of the present application;
[0033] Fig.15 is a schematic diagram of a video generating device according to an embodiment of the present application;
[0034] Fig.16 is a schematic diagram of a video generating device according to an embodiment of the present application;
[0035] Fig.17 It is a structural block diagram of a computer terminal according to an embodiment of the present application. DETAILED DESCRIPTION
[0036] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.
[0037] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0038] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following explanations:
[0039] Diffusion Model: The diffusion model gradually removes noise from Gaussian noise and generates the required image or video through multiple iterations. It is currently a better image or video generation model in deep learning.
[0040] Virtual Try-on: Virtual Try-on is a virtual fitting technology that aims to simulate the effect of real fitting and naturally apply the appearance of clothing to the target person. Virtual fitting algorithms usually use computer vision technology, especially image or video generation technology. Virtual fitting involves aligning the target clothing with the target person's body and adjusting the appearance of the clothing according to the scene and the person's posture and movement to provide users with a close-to-real fitting experience.
[0041] Video Synthesis: Video synthesis refers to the use of computer vision algorithm models to generate new video sequences. Video synthesis can be divided into text-generated video, image-generated video, video-generated video, etc. according to different input conditions.
[0042] U-Net model: U-Net is a convolutional neural network architecture for image segmentation. It is widely used in fields such as medical image segmentation. Its structure is derived from the full convolutional network. U-Net is obtained by modifying and expanding the full convolutional neural network structure. The characteristic of U-Net is that it has a U-shaped network structure, including an encoder and a decoder. The encoder is used to extract image features, and the decoder is used to map the extracted features back to the original image size and generate segmentation results.
[0043] CILP encoder (Contrastive Language-Image Pre-training): CLIP encoder is a neural network model used for cross-modal learning between images and texts. It can map image and text information into a common feature space, thereby achieving semantic alignment between images and texts. CLIP encoder can be used to extract features from image information to obtain semantic information and feature representation of the image.
[0044] According to an embodiment of the present application, a video generation method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0045] Figure 1 Schematic diagram of the hardware environment of a virtual reality device according to a video generation method of an embodiment of the present application. Figure 1 As shown, the virtual reality device 104 is connected to the terminal 106, and the terminal 106 is connected to the server 102 through a network. The above-mentioned virtual reality device 104 is not limited to: virtual reality helmets, virtual reality glasses, virtual reality all-in-one machines, etc., and the above-mentioned terminal 104 is not limited to PCs, mobile phones, tablet computers, etc. The server 102 can be a server corresponding to a media file operator, and the above-mentioned network includes but is not limited to: a wide area network, a metropolitan area network or a local area network.
[0046] Optionally, the virtual reality device 104 of this embodiment includes: a memory, a processor, and a transmission device. The memory is used to store an application, which can be used to execute: obtaining the original video of the biological object and the image information of the object to be displayed; identifying the target object in the original video and generating a temporally smooth local focus; amplifying the original video using the temporally smooth local focus to obtain an amplified video; inputting the amplified video into the video generation model, and guiding the video generation model to generate the target video based on the temporally smooth local focus and image information, thereby solving the technical problem of poor video effect presented by the video virtual try-on and achieving the purpose of improving the video presentation effect of the video virtual try-on.
[0047] The terminal of this embodiment can be used to display an original video of a biological object and image information of an object to be displayed on a presentation screen of a virtual reality (VR) device or an augmented reality (AR) device, wherein the object to be displayed is an object to be displayed on the biological object; identify the target object in the original video and generate a temporally smooth local focus, wherein the temporally smooth local focus is used to characterize the position information of the target object in different video frames of the original video, and the target object is an object that has been displayed on the biological object; generate a target video based on the original video, the image information and the temporally smooth local focus, wherein the object to be displayed is displayed on the biological object in the target video, and send the target video to the virtual reality device 104, and the virtual reality device 104 displays it at the target placement position after receiving the target video.
[0048] Optionally, the virtual reality device 104 of this embodiment has an eye-tracking HMD (Head Mount Display) head display and an eye-tracking module that have the same functions as those in the above embodiment, that is, the screen in the HMD head display is used to display real-time images, and the eye-tracking module in the HMD is used to obtain the real-time movement trajectory of the user's eyeballs. The terminal of this embodiment obtains the user's position information and movement information in the real three-dimensional space through the tracking system, and calculates the three-dimensional coordinates of the user's head in the virtual three-dimensional space, as well as the user's field of view direction in the virtual three-dimensional space.
[0049] Figure 2 is a structural block diagram of a computing environment of a video generation method according to an embodiment of the present application, such as Figure 2 As shown, computing environment 201 includes multiple computing nodes (such as servers) running on a distributed network (shown as 210-1, 210-2, ..., in the figure). Different computing nodes contain local processing and memory resources, and end users 202 can remotely run applications or store data in computing environment 201. Applications can be provided as multiple services 220-1, 220-2, 220-3 and 220-4 in computing environment 201, representing services "A", "D", "E" and "H" respectively.
[0050] The end user 202 can provide and access services through a web browser or other software application on the client, and in some embodiments, the end user 202's provision and / or request can be provided to the entry gateway 230. The entry gateway 230 can include a corresponding agent to handle the provision and / or request for the service (one or more services provided in the computing environment 201).
[0051] Services are provided or deployed based on various virtualization technologies supported by the computing environment 201. In some embodiments, services can be provided based on virtual machine (VM)-based virtualization, container-based virtualization, and / or similar methods. Virtual machine-based virtualization can be to simulate a real computer by initializing a virtual machine, and execute programs and applications without directly contacting any actual hardware resources. While the virtual machine virtualizes the machine, according to container-based virtualization, a container can be started to virtualize the entire operating system (OS) so that multiple workloads can run on a single operating system instance.
[0052] In an embodiment based on container virtualization, several containers of a service can be assembled into a Pod (e.g., a Kubernetes Pod). Figure 2 As shown, service 220-2 can be equipped with one or more Pods 240-1, 240-2, ..., 240-N (collectively referred to as Pods). Pods can include a proxy 245 and one or more containers 242-1, 242-2, ..., 242-M (collectively referred to as containers). One or more containers in a Pod process requests related to one or more corresponding functions of the service, and the proxy 245 generally controls network functions related to the service, such as routing, load balancing, etc. Other services can also be equipped with similar Pods.
[0053] During operation, executing a user request from end user 202 may require invoking one or more services in computing environment 201, and executing one or more functions of one service may require invoking one or more functions of another service. Figure 2 As shown, service "A" 220-1 receives a user request from end user 202 from ingress gateway 230, service "A" 220-1 may call service "D" 220-2, and service "D" 220-2 may request service "E" 220-3 to perform one or more functions.
[0054] The computing environment described above can be a cloud computing environment, where the allocation of resources is managed by the cloud service provider, allowing the development of functions without considering the implementation, adjustment or expansion of servers. The computing environment allows developers to execute code in response to events without building or maintaining complex infrastructure. Services can be divided into a set of functions that can be automatically and independently scaled, rather than expanding a single hardware device to handle potential loads.
[0055] Under the above operating environment, this application provides Figure 3 It should be noted that the video generation method of this embodiment can be Figure 1 The AR / VR device of the illustrated embodiment is executed. Figure 3 FIG. 1 is a flow chart of a video generation method according to an embodiment of the present application. Figure 3 As shown, the method may include the following steps:
[0056] Step S302, obtaining the original video of the biological object and the image information of the object to be displayed, wherein the object to be displayed is the object to be displayed on the biological object.
[0057] The original video in the above steps can be a video obtained by shooting a biological object. One or more objects can be displayed on the biological object. At this time, the biological object can be understood as a model. The biological object here can be a living object such as a human being or an animal. In the embodiment of the present application, the biological object is taken as a human being for explanation.
[0058] The image information in the above steps may be an image obtained by photographing the item to be displayed. The item to be displayed here may be an item that is desired to be displayed on the biological object. The item may replace an item already on the biological object. The item here may be clothing, shoes, accessories, etc.
[0059] In an optional embodiment, when a user needs to perform a video virtual fitting, the user can shoot the user through a mobile device, camera, or webcam to obtain an original video. The user can also shoot the items to be displayed through the mobile device, or obtain images of the items to be displayed from the Internet or e-commerce platforms. Thus, the mobile device can process the original video and image to generate a target video.
[0060] In another optional embodiment, when a user needs to perform a video virtual fitting, the user uploads the original video and image to the cloud through a mobile device, and the cloud server processes the original video and image to generate a target video. The cloud server can return the generated target video to the mobile device, which is displayed to the user for viewing.
[0061] After obtaining the original video of the biological object, an object detection algorithm can be used to detect the original video to determine whether the original video contains the biological object. The object detection algorithm can be YOLO (an object detection algorithm) or Faster R-CNN (Faster Region-based Convolutional Neural Networks, Chinese name is Fast Regional Convolutional Neural Network), but is not limited thereto.
[0062] If it is determined that the original video contains a biological object, the original video is stored for subsequent video virtual try-on; if it is determined that the original video does not contain a biological object, the original video is not saved, and the user is prompted that the original video is invalid, prompting the user to upload or shoot the original video containing the biological object again.
[0063] Step S304, identifying the target object in the original video and generating a temporally smoothed local focus, wherein the temporally smoothed local focus is used to characterize the position information of the target object in different video frames of the original video, and the target object is the object currently loaded on the biological object as a carrier.
[0064] The target item in the above steps is the item currently loaded on the biological object, which is of the same type as the item to be displayed. The position of the current target item on the biological object is the same as the target position where the item to be displayed needs to be displayed.
[0065] The temporally smoothed local focusing in the above steps can be obtained by combining the detection frames of the target objects in different video frames in the original video in chronological order, so that the temporally smoothed local focusing can not only reflect the correlation between different video frames, but also highlight the characteristics of the target objects in the same video frame. The above detection frames can be obtained by detecting the original image through the target detection algorithm, and each detection frame consists of center coordinates and size information. The temporally smoothed local focusing can accurately extract the position of the target object on the biological object at different times, and provide position guidance for the loading of the objects to be displayed.
[0066] In an optional embodiment, target detection algorithms such as YOLO, SSD (Single Shot MultiBox Detector), and Faster R-CNN can be used to detect target objects in different video frames of the original video, and the target object is identified and framed using the target detection algorithm to obtain a detection frame of the target object. The video frames containing the detection frame are combined in chronological order to obtain a video in which the target objects in different video frames are marked with the detection frame, and the video is used as a local focus with temporal smoothing.
[0067] In another optional embodiment, since the detection frame identified by the target detection algorithm sometimes cannot cover all parts of the target object, in order to avoid such a problem, after the detection frame is detected, the size of the detection frame can be expanded according to predefined rules to ensure that the bounding box covers the entire target object, and the video after the expanded detection frame is used as a temporally smooth local focus.
[0068] Temporally smoothed local focusing can provide a stable close-up of the area where the target object is located, while accurately describing the target position of the object to be displayed, improving the ability to maintain the information details of the target object, making the display effect more realistic when the object to be displayed is loaded onto a biological object.
[0069] Step S306, amplify the original video using temporally smoothed local focusing to obtain an amplified video.
[0070] The enlarged video in the above step is a video obtained by enlarging the original video using temporally smoothed local focusing.
[0071] In an optional embodiment, the original video is cropped according to the detection frame in the temporally smoothed local focus, and only the corresponding area in the detection frame in each video frame is retained to obtain a cropped video.
[0072] Since the size of the target object in each video frame of the original video may change, the cropped video may store the problem of different video frame sizes. Therefore, the cropped video is enlarged to obtain an enlarged video in which each video frame has a moderate size and the area where the target object is located in the video frame is centered. Specifically, a preset target resolution can be set as the standard for enlargement. According to the preset target resolution, some pixels are added to the edge of the video frame to fill the video frame so that it reaches the preset target resolution, thereby obtaining an enlarged video.
[0073] Step S308, inputting the enlarged video into a video generation model, and guiding the video generation model to generate a target video based on the temporally smoothed local focus and image information, wherein the object to be displayed is displayed on the biological object in the target video.
[0074] The video generation model in the above steps is a model used to process the enlarged video and generate the target video. It can be a U-Net model, a SegNet (Semantic Segmentation Network) model, or a FCN (Fully Convolutional Network), but is not limited to this.
[0075] In an optional embodiment, the enlarged video can be converted into a tensor form to be input into the video generation model. During the conversion process, TensorFlow (a deep learning framework) can be used to convert the enlarged video into a tensor and input into the video generation model.
[0076] The U-Net model can be selected as the video generation model. The U-Net model is a diffusion model. The information of the temporally smooth local focus is encoded and added to the video generation model as a position code to reduce the impact of slight jitter or jump of the temporally smooth local focus on the target video. The features in the image information are extracted and injected into the video generation model to provide information about the object to be displayed, so as to achieve the replacement of the target object and the object to be displayed on the biological object. The video generation model processes the enlarged video and generates a target video showing the object to be displayed on the biological object under the guidance of the temporally smooth local focus and image information.
[0077] In an embodiment of the present application, an original video of a biological object and image information of an object to be displayed are obtained; the target object in the original video is identified to generate a temporally smooth local focus; the original video is amplified using the temporally smooth local focus to obtain an amplified video; the amplified video is input into a video generation model, and the video generation model is guided to generate a target video based on the temporally smooth local focus and image information. It is easy to notice that a stable close-up shot of the area where the target object is located in the original video is extracted by temporally smooth local focus, so as to achieve the purpose of accurately locating the target object, and the original video is amplified using temporally smooth local focus, thereby amplifying the area where the target object is located, better retaining the details of the target object, and providing a more accurate reference for the placement and display details of the object to be displayed. Further, under the guidance of temporally smooth local focus and image information, a target video is generated based on the amplified video, thereby improving the presentation effect of the target video, thereby solving the technical problem of poor video effect presented by the video virtual try-on.
[0078] Optionally, the target object in the original video is identified to generate a temporally smooth local focus, including: respectively identifying the target object in multiple video frames of the original video to obtain initial display areas of the multiple video frames, wherein the initial display areas of different video frames are used to characterize position information of the target object in different video frames; expanding the initial display areas of the multiple video frames to obtain target display areas of the multiple video frames; and combining the target display areas of the multiple video frames according to the time sequence of the multiple video frames to obtain a temporally smooth local focus.
[0079] The initial display area in the above step is the area within the minimum bounding box obtained by intercepting the target item on the biological object in multiple video frames. For example, when the biological object is a person and the target item is clothing, the initial display area is the area within the minimum bounding box of the clothing worn by the person.
[0080] The target display area in the above steps is the area obtained by expanding the initial display area. The purpose of the expansion is to ensure that the target display area can fully cover the range of the target object without missing the boundaries and details of the target object. When expanding, the coordinates of the bounding box can be expanded according to preset rules. The preset rules can be preset according to actual application conditions, and the preset rules are not limited here.
[0081] In an optional embodiment, when identifying target objects in multiple video frames of an original video, the multiple video frames of the original video may be preprocessed, and the specific steps of the preprocessing may include but are not limited to image enhancement, noise removal, normalization, etc. After appropriate preprocessing, the performance of the target detection algorithm can be improved and the target object can be better identified. Use target detection algorithms such as the YOLO algorithm, the Faster R-CNN algorithm, and the SSD (Single Shot MultiBox Detector, the Chinese name is Single Shot MultiBox Detector) algorithm to extract features from multiple video frames of the original video, and identify the target object based on the extracted feature information, and output the initial display area of the target object in the video frame in the form of a rectangular bounding box.
[0082] The initial display area is then expanded to ensure that the expanded display area can cover all the boundaries and details of the target object. When expanding, a proportional expansion method can be used to expand the width and height of the initial display area boundary box by a certain proportion to obtain the target display area, ensuring that the overall shape of the target object is completely contained in the target display area. The expansion ratio can be determined based on experience, reference materials or actual application conditions, and the specific expansion ratio is not limited here.
[0083] A pixel-level expansion method can also be used to add a certain number of pixels around the initial display area boundary box to obtain a target display area to ensure that the boundaries and details of the target object are not blocked or truncated. The number and position of added pixels can also be determined based on experience, reference materials or actual application conditions, and the specific number and position of added pixels are not limited here.
[0084] Alternatively, a boundary detection-based expansion method may be used, where an edge detection algorithm is used to detect the boundary of the target object, and based on the detected boundary of the target object, the boundary box of the initial display area is expanded outward to obtain the target display area. However, the method of expanding the initial display area outward may not be limited to this.
[0085] Afterwards, the target display areas of multiple video frames are combined to obtain an arrangement order of the multiple video frames according to the time sequence from first to last. The video frames correspond to the target display areas. According to the arrangement order of the multiple video frames, the target display areas of the multiple video frames are arranged to obtain a local focus with smooth time sequence.
[0086] Optionally, target objects in multiple video frames of the original video are identified respectively to obtain initial display areas of the multiple video frames, including: determining posture information of biological objects in the original video; based on the posture information and the type of objects to be displayed, target objects in multiple video frames of the original video are identified respectively to obtain initial display areas of the multiple video frames.
[0087] The posture information in the above steps is information used to describe the posture of the biological object in the original video, which can be in the form of a posture image, a posture vector, or a key point coordinate. The key point can be a key position of a body part, such as the head, shoulder, elbow, knee, etc., or a key position in a specific action, such as the position where the finger touches the object, the position where the toe touches the ground, etc., but the form of the posture information is not limited to this. When the posture of the biological object is different, the shape and size of the target object on the biological object in the video frame may also change, resulting in a change in the initial display area. Therefore, the posture information can be used as auxiliary information to determine the initial display area.
[0088] The type of the object to be displayed in the above step can be used to determine the initial display area. The type of the object to be displayed is usually the same as the type of the target object. Since the types are the same, the object to be displayed and the target object can be replaced on the biological object.
[0089] In an optional embodiment, when determining the posture information of a biological object in an original video, a key point detection algorithm such as the Harris corner detection algorithm (Chinese name: Harris Corner Detection Algorithm), SIFT (Scale-Invariant Feature Transform, Chinese name: Scale-Invariant Feature Transform) algorithm, etc. may be used to detect the key point positions of the biological object in the video frame, and determine the posture information of the biological object through the key point positions; a posture recognition algorithm may also be used to identify the action and posture information of the biological object in the video frame; a motion tracking algorithm may also be used to track the motion trajectory of the biological object in the video, thereby inferring the posture information of the biological object, but the method for determining the posture information of the biological object may not be limited to this.
[0090] Then, based on the posture information and the type of the object to be displayed, the target object in the multiple video frames of the original video is identified, and the mapping relationship between the type of the object to be displayed and the approximate initial display area is predetermined.
[0091] During the recognition process, the approximate range of the initial display area is determined according to the type of the item to be displayed and the mapping relationship, and then the approximate initial display area is corrected according to the posture information to obtain the initial display area. For example, if the type of the item to be displayed is earrings, the approximate initial display area is the area around the human ear and below the ear. When the posture information is a large head tilt, the position of the earrings will be different from the position of the human body when it is static. The approximate initial display area should be rotated to obtain the initial display area that conforms to the posture information.
[0092] Optionally, target display areas of multiple video frames are combined according to the time sequence of the multiple video frames to obtain a local focus with smooth timing, including: combining target display areas of multiple video frames according to the time sequence of the multiple video frames to obtain a local focus with initial smooth timing; smoothing the initial smooth timing local focus to obtain a local focus with smooth timing.
[0093] The above steps are an intermediate concept in the process of determining the local focus of temporal smoothness, which can be obtained by combining the target display areas of multiple video frames and can be used to determine the local focus of temporal smoothness.
[0094] In an optional embodiment, multiple video frames are sorted in order of time from first to last, and then the sorted multiple video frames are combined to obtain an initial time-series smoothed local focus. The initial time-series smoothed local focus is then smoothed, the initial time-series smoothed local focus is corrected using a Kalman filter, and then the initial time-series smoothed local focus after the Kalman filter processing is further filtered using a low-pass filter to obtain a time-series smoothed local focus.
[0095] The purpose of using Kalman filtering for smoothing is to smooth the local focus of the obtained time series smoothing to achieve a change effect similar to the natural movement of the camera. Moreover, after the local focus of the initial time series smoothing is corrected using Kalman filtering, the high-frequency jitter caused by abnormal conditions can be further filtered out using a low-pass filter for processing.
[0096] Optionally, the original video is magnified using temporally smoothed local focusing to obtain an enlarged video, including: processing the original video based on the display area of the target object in the original video to obtain a first video, wherein the first video does not contain the target object; and magnifying the first video using temporally smoothed local focusing to obtain an enlarged video.
[0097] The display area in the above steps is the area on the biological object where the target item is located in the video frame of the original video. For example, when the target item is a top, the display area is the area of the upper body of the human body; when the target item is a pair of pants, the display area is the area of the lower body of the human body; when the target item is a dress, the display area is the area of the entire body of the human body.
[0098] In an optional embodiment, a target detection algorithm may be used to detect the target object in the video frame of the original video and determine the display area of the target object. According to the display area of the target object, a foreground segmentation algorithm is used to separate the target object and the background in the video frame, and a background filling technique is used to fill the position of the target object with the background color or background content to achieve the removal of the target object. Then, a repair algorithm is used to repair the traces or distortions left after the removal of the target object, and a first video without the target object is obtained.
[0099] In addition, the first video can also be determined by using a mask. Create a mask image with the same size as the video frame of the original video. In the mask image, set the pixel value of the display area to 1 and the pixel value of other areas to 0 to mark the area to be masked. Combine the video frame of the original video with the mask image, and mask the display area of the target object in the original video according to the pixel value of the mask to obtain the first video.
[0100] The first video is cropped according to the detection frame of the temporally smoothed local focus to obtain a video that only includes the display area, does not include other parts outside the display area, and has no target object in the display area. The cropped first video is enlarged with reference to the filling position and number of pixels when the temporally smoothed local focus is enlarged to obtain an enlarged video.
[0101] Optionally, the enlarged video is input into a video generation model, and the video generation model is guided to generate a target video based on the temporally smooth local focus and image information, including: extracting features of the temporally smooth local focus to obtain temporally smooth local focus features, extracting features of the enlarged video to obtain object features, and extracting features of the image information to obtain item features; inputting the object features into the video generation model, and guiding the video generation model to generate a predicted video based on the temporally smooth local focus features and item features.
[0102] The temporally smoothed local focusing features in the above steps may be vector representations for describing the temporally smoothed local focusing, and may include but are not limited to size features of the temporally smoothed local focusing, coordinate features of the temporally smoothed local focusing, and the like.
[0103] The object features in the above steps can be vector representations used to describe the enlarged video, can provide area information of the target object to guide the placement of the object to be displayed, and can also provide background information within the target object display area, which may include but is not limited to noise features, mask features of the target object display area, etc.
[0104] The object features in the above steps may include but are not limited to object category, color information, texture features, detail features, etc., which are used to guide the generation of the prediction video.
[0105] The predicted video in the above steps is a video generated by the video generation model. In the predicted video, the object to be displayed is located on the biological object, but the area displayed by the predicted video is smaller than the original video, and the displayed area is only the area where the target object is located.
[0106] In an optional embodiment, a video processing library may be used to extract features of the temporally smoothed local focus to obtain size features and coordinate features of the temporally smoothed local focus, thereby determining the temporally smoothed local focus features.
[0107] When extracting features from an enlarged video, inter-frame differences can be performed on consecutive frames of the video to calculate the differences between adjacent frames, extract the noise features of the enlarged video, and extract the mask features used to cover the display area mask based on the display area of the target object, thereby obtaining the object features of the enlarged video.
[0108] In the process of extracting features from image information, the VGG (Visual Geometry Group) model can be used to extract features of the objects to be displayed in the image information, and obtain the color histogram, texture features, shape features, etc. of the objects to be displayed, and obtain the object features. The CLIP encoder can also be used to process the image information to capture high-level semantic information of the objects to be displayed, such as the category of the object, color information, texture features, etc., and the U-Net model can be used to process the image information to extract the detailed features of the objects to be displayed, and the extracted features can be uniformly represented in the form of vectors to obtain the object features.
[0109] The object features extracted and processed from the enlarged video can provide the target position where the object to be displayed should be placed, and the object features are input into the video generation model. In addition, the object features can provide feature information of the object to be displayed, and in order to smooth the input form of the object features, it is necessary to introduce temporally smoothed local focus features, so that the video generation model generates a predicted video under the guidance of the object features and the temporally smoothed local focus features.
[0110] In the generated predicted video, the object to be displayed is located on the biological object, but since the enlarged video only contains the area related to the target object in the original video, it is also necessary to fuse the predicted video with the original video to increase the background information outside the area where the object to be displayed is located. In order to make the predicted video and the original video merge naturally, the predicted video can be processed with a Gaussian kernel first, and then fused with the original video to obtain the target video.
[0111] Optionally, feature extraction is performed on the local focus of temporal smoothing to obtain local focus features of temporal smoothing, including: encoding tunnel information of the local focus of temporal smoothing to obtain a one-dimensional position code; and processing the one-dimensional position code using a linear mapping and an activation function to obtain local focus features of temporal smoothing.
[0112] The tunnel information in the above steps may be the size of the video frame of the original video from which the temporally smoothed local focus is extracted, the central coordinates of the temporally smoothed local focus, the size of the temporally smoothed local focus, and the like.
[0113] The one-dimensional position coding in the above step is a coding obtained by assigning a unique position coding to each element in the sequence, and can be obtained by encoding the local focused tunnel information with temporal smoothing.
[0114] The linear mapping in the above steps refers to the process of mapping the input data to the output data through linear transformation, which is used to realize the linear transformation and feature extraction of the input data. The activation function is a nonlinear function in the neural network, which is used to introduce nonlinear factors and increase the expression and fitting ability of the model.
[0115] In an optional embodiment, the tunnel information of the temporally smoothed local focus may be in the form of a triplet, including the size of the video frame of the original video for extracting the temporally smoothed local focus, the center coordinates of the temporally smoothed local focus, and the size of the temporally smoothed local focus, and then the triplet is encoded into a one-dimensional position code.
[0116] Then, the one-dimensional position code is taken as input, and through a linear mapping operation, the one-dimensional position code is multiplied by the weight matrix and a bias term is added to obtain the output after the linear transformation. The activation function is used to activate the output after the linear transformation to obtain a local focus feature with smooth time series. Among them, the activation function can be a Sigmoid function (SigmoidActivation Function, the Chinese name is S-type function), a Tanh function (Hyperbolic Tangent ActivationFunction, the Chinese name is hyperbolic tangent function) or a ReLU function (Rectified Linear Unit ActivationFunction, the Chinese name is a corrected linear unit function), but is not limited to this. Using an activation function to process the one-dimensional position code can increase the expressive power of the local focus feature with smooth time series by introducing nonlinear factors.
[0117] Optionally, feature extraction is performed on the enlarged video to obtain object features, including: obtaining a noise video corresponding to the enlarged video; determining a mask video corresponding to the display area; and feature extraction is performed on the noise video, the mask video and the first video to obtain object features.
[0118] The noise video in the above steps is a video obtained by processing the enlarged video with Gaussian white noise. Gaussian white noise is a random signal with uniform power spectrum density, which is used to simulate the random noise existing in the enlarged video. The mask video is a video obtained by performing a mask operation on the display area in each video frame in the enlarged video.
[0119] In an optional embodiment, Gaussian white noise is a normally distributed random number with a mean of 0 and a variance of 1, and Gaussian white noise can be generated by a random number generator. The generated Gaussian white noise is superimposed on each video frame of the amplified video to simulate the noise existing in the amplified video, and the video frames are combined in chronological order to obtain a noise video.
[0120] When determining the mask video corresponding to the display area, first determine the display area of the target object in the original video, generate a mask image of the same size as the original video, and the position pixels within the display area in the mask image are 1, and the position pixels outside the display area are 0. Apply the mask image to the enlarged video, perform a pixel-by-pixel bitwise AND operation on the mask image and each frame of the enlarged video to obtain the mask video.
[0121] Feature extraction is performed on the noise video, the mask video and the first video to obtain object features that can describe the enlarged video.
[0122] Optionally, feature extraction is performed on the noise video, the mask video and the first video to obtain object features, including: using a posture encoder to extract features of posture information of the biological object in the original video to obtain posture features; performing feature extraction on the noise video, the mask video and the first video respectively to obtain noise features corresponding to the noise video, mask features corresponding to the mask video, and target features corresponding to the first video; splicing the noise features, the mask features and the target features to obtain a first spliced feature; and adding the first spliced feature to the posture feature to obtain the object feature.
[0123] The posture encoder in the above steps is a model for converting biological posture information into a specific coding representation, which can be a key point encoder, a skeleton encoder, or a variational autoencoder, but is not limited to these.
[0124] The posture feature in the above steps refers to a representative feature representation extracted from the posture information, which may include but is not limited to key point positions, skeleton structure motion trajectories, angles and directions, etc.
[0125] The first spliced feature in the above step is a feature obtained by splicing the noise feature, the mask feature and the target feature, which can be in the form of a tensor, but is not limited thereto. The noise feature is a feature extracted from the noise video, the mask feature is a feature extracted from the mask video, and the target feature is a feature extracted from the first video. The noise feature, the mask feature and the target feature are all used to generate the first spliced feature.
[0126] In an optional embodiment, in order to assist the generation process of the target video, enhance the guidance of the movement of the biological object in the target video, and further improve the fidelity of the target video, it is necessary to extract posture features from the posture information as additional control information. The posture information of the biological object in the original video is encoded by a posture encoder composed of multiple convolutional layers to obtain posture features.
[0127] The noise feature may be a video frame in a noise video, the mask feature may be each mask video frame in a mask video, and the target feature may be a mask that blocks the display area of the target item in the first video. Noise data is obtained from the noise video, each mask video frame is obtained from the mask video, and a mask that blocks the display area of the target item is obtained from the first video. The noise data, each mask video frame, and the mask that blocks the display area of the target item are encoded and converted into a tensor form, and spliced according to the channel dimension to obtain a first splicing feature. Exemplarily, the first splicing feature may include 9 channels, of which 4 channels are mask video frames, 4 channels are noise data, and 1 channel is a mask that blocks the display area of the target item. However, the above examples are only examples, and the number of channels in actual applications may not be limited to this.
[0128] Then, the first splicing feature can be added to the posture feature, and the posture feature can be added to the first splicing feature as a latent space feature to form an object feature that can be used as an input of a video generation model.
[0129] Optionally, feature extraction is performed on the image information to obtain object features, including: using an image encoder to extract features from the image information to obtain image features of the image information; using an image diffusion model to extract features from the image information to obtain object texture features of the image information; wherein the object features include image features and object texture features.
[0130] The image encoder in the above steps is an encoder for extracting image features from image information, and the image features may include but are not limited to color features, shape features, edge features, size features, contrast features, etc. The image encoder may be a CLIP encoder, a ViT (Vision Transformer), or a BERT (Bidirectional Encoder Representations from Transformers), but is not limited thereto.
[0131] The image diffusion model in the above steps is a model for extracting object texture features from image information. Both object texture features and image features can describe image information, but object texture features are more fine-grained than image features, focusing on describing image details, such as texture, spots, corners, subtle structures, etc. The image diffusion model can be a U-Net model or an FCN model, but is not limited thereto.
[0132] In an optional embodiment, the object features include image features and object texture features. When determining the image features, a CLIP encoder can be selected as an image encoder, and the image information is processed using the CLIP encoder. The CLIP encoder extracts high-level feature representations of image information through a multi-layer convolutional neural network, uses high-dimensional vectors to reflect the abstract expression of image information at different levels, and normalizes and normalizes the high-level features to output image features of the image information.
[0133] When determining the texture features of an object, the U-Net model can be selected as the image diffusion model. In order to distinguish it from the U-Net model used as a video generation model, the U-Net model used as a video generation module can be called the main U-Net, and the U-Net model used as an image diffusion model to extract the texture features of an object can be called the reference U-Net. The U-Net model is a convolutional neural network used for image segmentation, which usually includes an encoder and a decoder, and can effectively extract the detailed features of an image. When the U-Net model is used to process image information, in the encoder part, the input image information is subjected to feature extraction and downsampling through a series of convolutional layers and pooling layers, gradually reducing the spatial size of the image and increasing the number of channels to obtain the feature representation of the image details. At the end of each layer of the encoder, the feature map of the layer is saved and passed to the decoder part. The decoder receives the feature map from the encoder, reconstructs and refines the features through a series of convolutional layers and upsampling layers, gradually restores the spatial size of the image and reduces the number of channels, and obtains the texture features of the object.
[0134] Optionally, the object features are input into a video generation model, and the video generation model is guided to generate a predicted video based on the temporally smooth local focus features and object features, including: using an environment encoder to extract features from the first video to obtain environment features, wherein the environment features are used to characterize the content in the original video other than the target object; the object features are input into a video generation model, and the video generation model is guided to generate a predicted video based on the temporally smooth local focus features, environment features, and object features.
[0135] The environment encoder in the above steps is a neural network model used to capture environmental or background information. In video processing, the environment encoder can be used to extract global context information in each video frame.
[0136] The environmental features in the above steps are global context information features in the original video, which can be used to generate a reasonable background in the display area.
[0137] In an optional embodiment, since extracting temporally smooth local focus from the original video will result in partial loss of environmental context, it increases the difficulty of generating a reasonable background in the display area when generating the predicted video. Therefore, an environmental encoder is used to extract global features other than temporally smooth local focus, and integrate them into the video generation model to enhance the generation of the background.
[0138] The environment encoder may be composed of a frozen CLIP encoder and a learnable linear mapping layer, wherein the first video is encoded using the CLIP encoder to capture overall information about the environment, and the overall information about the environment output is fine-tuned through the learnable linear mapping layer to obtain environment features.
[0139] The environmental features are injected as keys and values into the main U-Net, that is, the video generation model. After the object features are input into the main U-Net, they are guided by the video generation model to generate predicted videos together with the temporally smoothed local focus features and object features.
[0140] Optionally, the video generation model includes: multiple spatial attention modules and multiple temporal attention modules that are alternately connected; inputting object features into the video generation model, and guiding the video generation model to generate a predicted video based on temporally smoothed local focus features, environmental features, and object features, including: using any spatial attention module to perform attention processing on input features, environmental features, and object features to obtain a first attention feature output by any spatial attention module, wherein, when any attention module is the first module in the video generation model, the input feature is an object feature, and the first attention feature output by any attention module is output to a temporal attention module connected to any attention module; using any temporal attention module to perform attention processing on the input first attention feature and temporally smoothed local focus features to obtain a second attention feature output by any temporal attention module; reconstructing the second attention feature output by the last temporal attention module among multiple temporal attention modules to generate a predicted video.
[0141] The spatial attention module in the above steps is a module in the video generation model used to enhance the video generation model's attention to spatial information. The spatial attention module mainly focuses on the changes and correlations of the input data in the spatial dimension. By learning the spatial weights, the model can better capture the relationship between spatial information and improve the model's performance in the spatial dimension.
[0142] The temporal attention module in the above steps is a module in the video generation model used to enhance the video generation model's attention to temporal information. The temporal attention module mainly focuses on the changes and correlations of the input data in the temporal dimension. By learning the temporal weights, the model can better capture the dynamic features in the time series data and improve the performance of the model in the temporal dimension.
[0143] The input features in the above steps are used to input into the spatial attention module. When the spatial attention module is the first module in the video generation model, the input features are object features; when the spatial attention module is not the first module in the video generation model, the input features are the second attention features output by the previous temporal attention module adjacent to the spatial attention module.
[0144] The first attention feature in the above step is the feature obtained after the spatial attention module performs attention processing on the input features, environmental features and object features, and can be used as the input of the next temporal attention module adjacent to the spatial attention module.
[0145] The second attention feature in the above step is the feature obtained by the temporal attention module by performing attention processing on the input first attention feature and the temporally smoothed local focus feature, and can be used as the input of the next spatial attention module adjacent to the temporal attention module.
[0146] In an optional embodiment, Figure 4 is a schematic diagram of an optional video generation model according to an embodiment of the present application, such as Figure 4 As shown, the main U-Net, i.e., the video generation model 400, includes multiple spatial attention modules and multiple temporal attention modules that are alternately connected. The use of multiple spatial attention modules and multiple temporal attention modules that are alternately connected can enhance the spatiotemporal consistency of the generated predicted video.
[0147] The input features, environmental features and object features are injected into the spatial attention module, which performs self-attention and cross-attention processing on the input features, environmental features and object features to obtain the first attention feature, and outputs the first attention feature to the next temporal attention module adjacent to it. When the spatial attention module is the first module in the video generation model, the input feature is the object feature; when the spatial attention module is not the first module in the video generation model, the input feature is the second attention feature output by the previous temporal attention module adjacent to the spatial attention module.
[0148] The first attention feature and the temporally smoothed local focus feature are input into the temporal attention module, which performs self-attention processing on the first attention feature and the focus attention feature to obtain the second attention feature, and outputs the second attention feature to the next adjacent spatial attention module.
[0149] Injecting temporally smoothed local focus features into the temporal attention module helps improve the temporal consistency of predicted video generation. Since the bounding box in the temporally smoothed local focus extraction process is only calculated within a single frame, the relationship between frames is not considered, and the movement of the person and the camera may cause slight jitters or jumps in the temporally smoothed local focus. Therefore, compared with naturally captured videos, these jitters and jumps may cause the temporally smoothed local focus to look unnatural, increase the difficulty of temporal attention convergence, and lead to reduced temporal consistency in the generated predicted video. Therefore, the temporally smoothed local focus features can be added to the temporal attention module as position encoding. Due to the embedding of the temporally smoothed local focus features, the temporal attention module integrates the size and position information of the temporally smoothed local focus, which can prevent alignment errors caused by too large a span of the temporally smoothed local focus, thereby improving the temporal consistency of the predicted video generation.
[0150] In the last temporal attention module, the video frames with the objects to be displayed are spliced according to the generated second attention features to obtain the predicted video.
[0151] Optionally, any spatial attention module includes: a reference attention module and an environmental attention module; using any spatial attention module to perform attention processing on input features, environmental features and object features to obtain a first attention feature output by any spatial attention module, including: splicing the input features and the object texture features in the object features to obtain a second splicing feature; using the reference attention module to perform self-attention processing on the second splicing feature to obtain a self-attention feature; splicing the image features in the environmental features and the object features to obtain a third splicing feature; using the environmental attention module to perform cross-attention processing on the third splicing feature and the self-attention feature to obtain a first attention feature.
[0152] The reference attention module in the above steps is used to adjust the weights of the feature images according to the similarity between the object features and the item features in the input video generation model, thereby improving the performance of the model.
[0153] The environmental attention module in the above steps is used to adjust the weight of the feature image according to the relationship between the object features and the surrounding environment information, so as to better capture the contextual information in the image information.
[0154] The second concatenated feature in the above step is a feature obtained by concatenating the input feature and the item texture feature in the item feature, which is used to input into the reference attention module for self-attention processing. The weight of the input feature is adjusted according to the similarity between the input feature and the item texture feature to obtain the self-attention feature.
[0155] The third splicing feature in the above step is the feature obtained by splicing the environmental feature and the image feature, which is used to input into the environmental attention module for cross-attention processing, and the weight of the input feature is adjusted according to the context information to obtain the first attention feature.
[0156] In an optional embodiment, Figure 5 is a schematic diagram of an optional spatial attention module performing attention processing according to an embodiment of the present application, such as Figure 5 As shown, the spatial attention module 500 includes a reference attention module and an environment attention module. Taking the first spatial attention module as an example, when the biological object is a person, the input feature can be a person denoising feature map, when the target object and the object to be displayed are clothing, the object texture feature in the object feature can be a clothing detail feature map, and the image feature in the object feature can be a clothing image feature embedding.
[0157] Figure 5 The C in represents the splicing operation. The character denoising feature map input to the spatial attention module is expanded in the time dimension, that is, the frame number f dimension is added. In order to make the dimensions the same, the clothing detail feature map is also repeated f times and spliced with the character denoising feature map to obtain a second splicing feature. The spliced second splicing feature is flattened along the spatial dimension and input into the reference attention module of the spatial attention module 500. The reference attention module performs self-attention processing on the second splicing feature and retains the denoising feature part to obtain a self-attention feature.
[0158] Figure 5 The environment embedding in is also the environment feature. The clothing image feature embedding and the environment embedding are spliced to obtain the third spliced feature. The third spliced feature and the self-attention feature are both input into the environment attention module, which performs cross-attention processing to enhance the interactivity between features and obtain the first attention feature.
[0159] Optionally, any temporal attention module includes: a self-attention module; using any temporal attention module to perform attention processing on the input first attention feature and the temporally smoothed local focus feature to obtain a second attention feature output by any temporal attention module, including: splicing the temporally smoothed local focus feature and the first attention feature to obtain a fourth spliced feature; using the self-attention module to perform self-attention processing on the fourth spliced feature to obtain a second attention feature.
[0160] The fourth splicing feature in the above step is a feature obtained by splicing the temporally smoothed local focus feature and the first attention feature, which can be used to determine the second attention feature.
[0161] In an optional embodiment, the temporal attention module can perform self-attention processing on features between different frames at the same spatial position to ensure a smooth transition between frames.
[0162] Figure 6 is a schematic diagram of attention processing performed by an optional time attention module according to an embodiment of the present application, such as Figure 6 As shown, the temporally smoothed local focus feature is concatenated with the first attention feature output by the spatial attention module to obtain a fourth concatenated feature. The fourth concatenated feature is injected into the temporal attention module, and the self-attention module in the temporal attention module performs self-attention processing on the fourth concatenated feature to obtain a second attention feature.
[0163] Optionally, the predicted video is fused with the original video to obtain a target video, including: processing the predicted video using a Gaussian kernel to obtain a processed video; and fusing the processed video with the original video to obtain the target video.
[0164] The Gaussian kernel in the above steps is a kernel function used in image processing and pattern recognition. When fusing the predicted video and the original video, first using the Gaussian kernel to process the predicted video and then performing video fusion can improve the naturalness of the fusion, make the fused video clearer and smoother, and improve the quality and effect of the target video.
[0165] In an optional embodiment, when using a Gaussian kernel to process a predicted video, the predicted video is decomposed into frames, and each frame is processed using a Gaussian kernel function. During the processing, the parameters of the Gaussian kernel function, such as the standard deviation and the convolution kernel size, are adjusted to make the central area of the predicted video clear and the edge area increasingly blurred. The processed video frames are then fused with the video frames of the original video, and the fused video frames are recombined into a video to obtain a target video.
[0166] Optionally, the method also includes: obtaining first training data, wherein the first training data includes a first training image showing the first object and a second training image showing the second object; using the first training data to train a video generation model, a posture encoder, and an image diffusion model; obtaining second training data, wherein the second training data includes a first training video showing the first object and a second training video showing the second object; and using the second training data to train the video generation model and the environment encoder.
[0167] The first training data in the above steps is data used to train the video generation model, the posture encoder and the image diffusion model, and the first training data includes paired image data.
[0168] The first object in the above steps is a biological object, which may or may not have a target object, and the second object is an object to be displayed.
[0169] In an optional embodiment, the training phase can be divided into two stages. In the first stage, the model that needs to be trained does not include the temporal attention module, environmental encoder and tunnel embedding of the video generation model. In addition, the weights of the VAE encoder (VariationalAutoencoder, the Chinese name is variational autoencoder) and decoder, as well as the weights of the CLIP encoder are frozen, and only the parameters of the main U-Net, reference U-Net and gesture encoder are updated. The main U-Net is also the video generation model, and the reference U-Net is also the image diffusion model. In the first stage, the video generation model, gesture encoder and image diffusion model are trained.
[0170] Image fitting data may be obtained as the first training data, the image fitting data including a first training image showing a model and a second training image showing clothes or accessories, the first training image and the second training image being present in pairs. When obtaining the image fitting data, an open source dataset containing paired fitting data may be found from an open source dataset through a search engine or a dataset website and downloaded; training data may also be provided by a user, for example, image fitting data may be provided by a merchant on an e-commerce platform, but the method of obtaining the image fitting data may not be limited thereto.
[0171] The video generation model, posture encoder and image diffusion model are trained using image fitting data, so that the video generation model, posture encoder and image diffusion model can learn the ability to extract and retain object features, thereby achieving high-fidelity image-level object display.
[0172] In the second stage, video data is used for training, and only the parameters of the temporal attention module and the environment encoder are updated. The temporal attention module is located in the video generation model, so the second stage is the stage for training the video generation model and the environment encoder.
[0173] The video fitting data is obtained as the second training data. When obtaining the video fitting data, it is possible to cooperate with fashion brands or e-commerce platforms to obtain the video fitting data from merchants on the e-commerce platforms. The video fitting data can also be captured from major video platforms, but the method of obtaining the video fitting data is not limited to this. The video fitting data includes a first training video showing a model and a second training video showing clothes or accessories.
[0174] The temporal attention module and environment encoder of the video generation model are trained using video fitting data. The goal of this stage is to enable the video generation model and environment encoder to learn the information between video frames, while using the image-level object display capabilities learned in the first stage to generate predicted videos with high spatiotemporal consistency.
[0175] A preferred embodiment is described below. Figure 7 is a schematic diagram of an optional video generation method according to an embodiment of the present application, such as Figure 7 As shown, the temporally smoothed local focus 702 is magnified and converted into a channel tensor to obtain an enlarged video 704. The size of the enlarged video can be described as B×9×H×W, where B is the number of video frames of the enlarged video input each time, H is the length of the video frame of the enlarged video, W is the width of the video frame of the enlarged video, and 9 is the number of channels, but the number of channels is only an example. In actual application, other suitable values can be selected as the number of channels. C represents a splicing operation, which splices the enlarged video according to the channel dimension, and uses a posture encoder to extract posture features from the original video 706. + represents an element addition operation, which adds the enlarged video and posture features spliced according to the channel dimension to obtain object features. The object features are input into the main U-Net, that is, the video generation model 708.
[0176] The image information 712 is processed using the CILP encoder and the reference U-Net, that is, the image encoder and the image diffusion model 710. The image encoder is used to extract the overall image features in the image information 712, and the image diffusion model 710 is used to extract more detailed object texture features in the image information 712. The image features are input into the cross attention module in the image diffusion model 710 for attention processing and splicing with the object texture features. The self-attention module in the image diffusion model 710 inputs the spliced object features into the spatial attention module of the video generation model 708.
[0177] The environment encoder is used to extract the environment features in the temporally smoothed local focus 702, and the environment features are also injected into the spatial attention module of the video generation model 708.
[0178] In addition, after the temporally smoothed local focus 702 is corrected using the Kalman filter, the temporally smoothed local focus features are extracted from the temporally smoothed local focus 702, and the temporally smoothed local focus features are injected into the temporal attention module of the video generation model 708.
[0179] The input object features are processed using the video generation model 708, and a predicted video 714 is generated under the guidance of the object features, the environment features, and the temporally smoothed local focus features.
[0180] It is easy to notice that this method has good high-fidelity generation effect and detail preservation ability. It magnifies the temporally smoothed local focus 702 and exerts the ability to follow the details of the replaced clothing by magnifying the cropped area. While using the CILP encoder to extract the overall image features in the image information 712, the reference U-Net is used to extract more detailed object texture features, ensuring the ability to preserve the details of the replaced clothing, thereby generating high-fidelity fitting results with high detail preservation ability.
[0181] In addition, the use of an environment encoder to extract environmental features in the temporally smooth local focus 702 can provide a global context for the background around the clothing, thereby enhancing the generation of the background in the predicted video 714; the use of a posture encoder to extract the posture features of the character can enhance the guidance of the character's movement in the predicted video 714, thereby further improving the quality of the generated predicted video 714 and improving the fidelity of the predicted video 714.
[0182] In addition, the method has a strong adaptability to complex motions. It uses Kalman filter calculation to correct the temporally smoothed local focus 702, and injects the temporally smoothed local focus information into the temporal attention module of the video generation model 708. It can reduce the slight jitter and jump of the temporally smoothed local focus caused by the movement of the person and the camera, thereby maintaining the smoothness and continuity of the cropped video area, and significantly improving the temporal consistency of the generated predicted video 714, so that the video generation model 708 can cope with various relative motion relationships between cameras and people, such as the far and near motion of the person relative to the camera, the parallel motion of the person relative to the camera, the change of camera perspective, etc.
[0183] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.
[0184] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.
[0185] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), including several instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.
[0186] According to an embodiment of the present application, an embodiment of a video generation method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0187] Figure 8 is a flow chart of a video generation method according to an embodiment of the present application, such as Figure 8 As shown, the method may include the following steps:
[0188] Step S802, in response to the image input instruction acting on the operation interface, displaying the original video of the biological object and the image information of the object to be displayed on the operation interface, wherein the object to be displayed is an object to be displayed on the biological object;
[0189] Step S804, in response to the generation instruction acting on the operation interface, the target video is displayed on the operation interface, wherein the object to be displayed is displayed on the biological object in the target video, and the target video is a video generated based on the original video, image information and temporally smoothed local focus, and the temporally smoothed local focus is generated by identifying the target object in the original video, and the temporally smoothed local focus is used to characterize the position information of the target object in different video frames of the original video, and the target object is an object that has been displayed on the biological object.
[0190] The operation interface in the above steps refers to the interface for users to interact with computers or other devices, which can include graphical interfaces, text interfaces, etc.
[0191] The image input instructions in the above steps refer to instructions input by the user through the operation interface, which are used to control the video content and image content displayed on the operation interface.
[0192] The generation instruction in the above steps refers to the instruction issued by the user on the operation interface, which is used to generate the target video based on the original video, image information and temporally smoothed local focus.
[0193] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in the embodiments, as well as the application scenario and implementation process, but is not limited to the scheme provided in the embodiments.
[0194] According to an embodiment of the present application, an embodiment of a video generation method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0195] Fig. 9 is a flow chart of a video generation method according to an embodiment of the present application, such as Fig. 9 As shown, the method may include the following steps:
[0196] Step S902, obtaining the original video of the user and image information of the first clothing, wherein the first clothing is the clothing to be worn by the user;
[0197] Step S904, identifying the second clothing in the original video, and generating a temporally smoothed local focus, wherein the temporally smoothed local focus is used to characterize position information of the second clothing in different video frames of the original video, and the second clothing is clothing that the user has worn;
[0198] Step S906: Generate a target video based on the original video, the image information and the temporally smoothed local focus, wherein the user in the target video wears the first clothing.
[0199] The video fitting method can be applied to video virtual fitting scenarios of e-commerce platforms or clothing customization. In the video virtual fitting scenario, the original video contains a user wearing a second garment, and the image information contains a first garment that the user wants to virtually try on. After the second garment worn by the user in the original video is identified and a temporally smoothed local focus is generated, a virtual fitting video of the user wearing the first garment, i.e., the target video, can be generated based on the original video, the image information, and the temporally smoothed local focus.
[0200] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in the embodiments, as well as the application scenario and implementation process, but is not limited to the scheme provided in the embodiments.
[0201] According to an embodiment of the present application, a video generation method is also provided that can be applied to virtual reality scenarios such as virtual reality VR devices and augmented reality AR devices. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0202] Fig.10 FIG. 1 is a flow chart of a video generation method according to an embodiment of the present application. Fig.10 As shown, the method may include the following steps:
[0203] Step S1002, displaying the original video of the biological object and the image information of the object to be displayed on the presentation screen of the virtual reality VR device or the augmented reality AR device, wherein the object to be displayed is an object to be displayed on the biological object;
[0204] Step S1004, identifying the target object in the original video and generating a temporally smoothed local focus, wherein the temporally smoothed local focus is used to characterize the position information of the target object in different video frames of the original video, and the target object is an object that has been displayed on the biological object;
[0205] Step S1006, generating a target video based on the original video, the image information and the temporally smoothed local focus, wherein the object to be displayed is displayed on the biological object in the target video;
[0206] Step S1008: driving the VR device or AR device to display the target video.
[0207] The presentation screen in the above steps is the image or video content displayed on the virtual reality VR device or the augmented reality AR device. Optionally, in this embodiment, the above video generation method can be applied to a hardware environment consisting of a server and a virtual reality device. The original video of the biological object and the image information of the object to be displayed are displayed on the presentation screen of the virtual reality VR device or the augmented reality AR device. The server can be a server corresponding to the media file operator. The above network includes but is not limited to: a wide area network, a metropolitan area network or a local area network. The above virtual reality device is not limited to: a virtual reality helmet, a virtual reality glasses, a virtual reality all-in-one machine, etc.
[0208] Optionally, the virtual reality device includes: a memory, a processor, and a transmission device. The memory is used to store an application, which can be used to execute: displaying an original video of a biological object and image information of an object to be displayed on a presentation screen of a virtual reality VR device or an augmented reality AR device, wherein the object to be displayed is an object to be displayed on the biological object; identifying the target object in the original video and generating a temporally smooth local focus, wherein the temporally smooth local focus is used to characterize the position information of the target object in different video frames of the original video, and the target object is an object that has been displayed on the biological object; generating a target video based on the original video, image information, and temporally smooth local focus, wherein the object to be displayed is displayed on the biological object in the target video; and driving the VR device or AR device to display the target video. It should be noted that the above-mentioned video generation method applied in a VR device or an AR device of this embodiment may include Figure 3 The method of the illustrated embodiment is used to achieve the purpose of driving a VR device or an AR device to display a target video.
[0209] Optionally, the processor of this embodiment can call the application stored in the memory to execute the above steps through a transmission device. The transmission device can receive media files sent by the server through the network, and can also be used for data transmission between the processor and the memory.
[0210] Optionally, in a virtual reality device, a head-mounted display with eye tracking is provided, wherein the screen in the HMD is used to display the displayed video images, the eye tracking module in the HMD is used to obtain the real-time movement trajectory of the user's eyes, the tracking system is used to track the user's position information and motion information in the real three-dimensional space, and the computing processing unit is used to obtain the user's real-time position and motion information from the tracking system, and calculate the three-dimensional coordinates of the user's head in the virtual three-dimensional space, as well as the user's field of view direction in the virtual three-dimensional space, etc.
[0211] In an embodiment of the present application, a virtual reality device may be connected to a terminal, and the terminal and the server are connected through a network. The virtual reality device is not limited to: a virtual reality helmet, virtual reality glasses, a virtual reality all-in-one machine, etc. The terminal is not limited to a PC, a mobile phone, a tablet computer, etc. The server may be a server corresponding to a media file operator, and the network includes but is not limited to: a wide area network, a metropolitan area network or a local area network.
[0213] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in the embodiments, as well as the application scenario and implementation process, but is not limited to the scheme provided in the embodiments.
[0214] According to an embodiment of the present application, an embodiment of a video generation method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0215] Fig.11 is a flow chart of a video generation method according to an embodiment of the present application, such as Fig.11 As shown, the method may include the following steps:
[0216] Step S1102, acquiring the original video of the biological object and the image information of the object to be displayed by calling the first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter includes the original video and the image information, and the object to be displayed is an object to be displayed on the biological object;
[0217] Step S1104, identifying the target object in the original video and generating a temporally smoothed local focus, wherein the temporally smoothed local focus is used to characterize the position information of the target object in different video frames of the original video, and the target object is an object that has been displayed on the biological object;
[0218] Step S1106, generating a target video based on the original video, the image information and the temporally smoothed local focus, wherein the object to be displayed is displayed on the biological object in the target video;
[0219] Step S1108: output the target video by calling the second interface, wherein the second interface includes a second parameter, and a parameter value of the second parameter includes the target video.
[0220] The first interface in the above steps is an interface for acquiring the original video of the biological object and the image information of the object to be displayed, and the second interface refers to an interface for outputting the generated target video.
[0221] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in the embodiments, as well as the application scenario and implementation process, but is not limited to the scheme provided in the embodiments.
[0222] According to an embodiment of the present application, a video generating device for implementing the above embodiment is also provided. Fig.12 is a schematic diagram of a video generating device according to an embodiment of the present application, such as Fig.12 As shown, the device comprises:
[0223] An acquisition module 120, used to acquire the original video of the biological object and the image information of the object to be displayed, wherein the object to be displayed is the object to be displayed on the biological object;
[0224] A first generating module 122 is used to identify a target object in the original video and generate a temporally smoothed local focus, wherein the temporally smoothed local focus is used to characterize position information of the target object in different video frames of the original video, and the target object is an object currently loaded on the biological object as a carrier;
[0225] A magnification module 124, used to magnify the original video using temporally smoothed local focusing to obtain a magnified video;
[0226] The second generation module 126 is used to input the enlarged video into the video generation model, and guide the video generation model to generate a target video based on the temporally smoothed local focus and image information, wherein the object to be displayed is displayed on the biological object in the target video.
[0227] Optionally, the first generation module includes: an identification unit, used to identify the target objects in multiple video frames of the original video respectively, and obtain initial display areas of the multiple video frames, wherein the initial display areas of different video frames are used to characterize the position information of the target object in different video frames; an expansion unit, used to expand the initial display areas of the multiple video frames to obtain target display areas of the multiple video frames; a combination unit, used to combine the target display areas of the multiple video frames according to the time sequence of the multiple video frames to obtain a local focus with smooth temporal sequence.
[0228] Optionally, the recognition unit is also used to determine the posture information of the biological object in the original video; based on the posture information and the type of the object to be displayed, the target object in multiple video frames of the original video is identified respectively to obtain the initial display areas of the multiple video frames.
[0229] Optionally, the recognition unit is also used to determine the posture information of the biological object in the original video; based on the posture information and the type of the object to be displayed, the target object in multiple video frames of the original video is identified respectively to obtain the initial display areas of the multiple video frames.
[0230] Optionally, the combining unit is further used to combine the target display areas of the multiple video frames according to the time sequence of the multiple video frames to obtain a local focus with initial time sequence smoothness; and to smooth the local focus with initial time sequence smoothness to obtain a local focus with time sequence smoothness.
[0231] Optionally, the magnification module includes: a processing unit, used to process the original video based on the display area of the target object in the original video to obtain a first video, wherein the first video does not contain the target object; and an amplification unit, used to amplify the first video using time-smoothed local focusing to obtain an amplified video.
[0232] Optionally, the second generation module includes: an extraction unit, used to perform feature extraction on the temporally smoothed local focus to obtain temporally smoothed local focus features, perform feature extraction on the enlarged video to obtain object features, and perform feature extraction on the image information to obtain item features; a generation unit, used to input the object features into the video generation model, and guide the video generation model to generate a predicted video based on the temporally smoothed local focus features and item features; a fusion unit, used to fuse the predicted video with the original video to obtain a target video.
[0233] The extraction unit is also used to encode the local focused tunnel information with temporal smoothness to obtain a one-dimensional position code; and the one-dimensional position code is processed by using a linear mapping and an activation function to obtain a local focused feature with temporal smoothness.
[0234] The extraction unit is also used to obtain a noise video corresponding to the enlarged video; determine a mask video corresponding to the display area; and perform feature extraction on the noise video, the mask video and the first video to obtain object features.
[0235] The extraction unit is also used to use a posture encoder to perform feature extraction on the posture information of the biological object in the original video to obtain posture features; perform feature extraction on the noise video, the mask video and the first video respectively to obtain noise features corresponding to the noise video, mask features corresponding to the mask video, and target features corresponding to the first video; splice the noise features, the mask features and the target features to obtain a first spliced feature; and add the first spliced feature to the posture feature to obtain an object feature.
[0236] The extraction unit is also used to extract features from the image information using an image encoder to obtain image features of the image information; and to extract features from the image information using an image diffusion model to obtain object texture features of the image information; wherein the object features include image features and object texture features.
[0237] The generation unit is also used to use the environment encoder to extract features from the first video to obtain environment features, wherein the environment features are used to characterize the content in the original video other than the target object; the object features are input into the video generation model, and the video generation model is guided to generate a predicted video based on the temporally smoothed local focus features, environment features and object features.
[0238] The generation unit is also used to use any spatial attention module to perform attention processing on input features, environmental features and object features to obtain a first attention feature output by any spatial attention module, wherein, when any attention module is the first module in the video generation model, the input feature is an object feature, and the first attention feature output by any attention module is output to a temporal attention module connected to any attention module; use any temporal attention module to perform attention processing on the input first attention feature and the temporally smoothed local focus feature to obtain a second attention feature output by any temporal attention module; reconstruct the second attention feature output by the last temporal attention module among multiple temporal attention modules to generate a predicted video.
[0239] The generation unit is also used to splice the input features and the texture features of the item in the item features to obtain a second spliced feature; use the reference attention module to perform self-attention processing on the second spliced feature to obtain a self-attention feature; splice the environment features and the image features in the item features to obtain a third spliced feature; use the environment attention module to perform cross-attention processing on the third spliced feature and the self-attention feature to obtain a first attention feature.
[0240] The generation unit is also used to splice the temporally smoothed local focus feature and the first attention feature to obtain a fourth spliced feature; and use the self-attention module to perform self-attention processing on the fourth spliced feature to obtain a second attention feature.
[0241] The fusion unit is also used to process the predicted video using the Gaussian kernel to obtain processed features; and fuse the processed features with the original video to obtain the target video.
[0242] The second generation module is also used to obtain first training data, wherein the first training data includes a first training image showing the first object and a second training image showing the second object; use the first training data to train the video generation model, the posture encoder and the image diffusion model; obtain second training data, wherein the second training data includes a first training video showing the first object and a second training video showing the second object; use the second training data to train the video generation model and the environment encoder.
[0243] It should be noted that the acquisition module 120, the first generation module 122, the amplification module 124, and the second generation module 126 correspond to steps S302 to S308 in the embodiment, and the modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors, and the above modules can also be run in the AR / VR device provided in the embodiment as part of the device.
[0244] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in the embodiments, as well as the application scenario and implementation process, but is not limited to the scheme provided in the embodiments.
[0245] According to an embodiment of the present application, a video generating device for implementing the above embodiment is also provided. Fig.13 is a schematic diagram of a video generating device according to an embodiment of the present application, such as Fig.13 As shown, the device comprises:
[0246] A first display module 130, for responding to an image input instruction acting on the operation interface, and displaying the original video of the biological object and the image information of the object to be displayed on the operation interface, wherein the object to be displayed is an object to be displayed on the biological object;
[0247] The second display module 132 is used to respond to the generation instruction acting on the operation interface and display the target video on the operation interface, wherein the object to be displayed is displayed on the biological object in the target video, and the target video is a video generated based on the original video, image information and time-smoothed local focus, and the time-smoothed local focus is generated by identifying the target object in the original video, and the time-smoothed local focus is used to characterize the position information of the target object in different video frames of the original video, and the target object is an object that has been displayed on the biological object.
[0248] It should be noted that the first display module 130 and the second display module 132 correspond to steps S802 to S804 in the embodiment, and the examples and application scenarios implemented by the modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above embodiments. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors, and the above modules can also be run in the AR / VR device provided in the embodiment as part of the device.
[0249] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in the embodiments, as well as the application scenario and implementation process, but is not limited to the scheme provided in the embodiments.
[0250] According to an embodiment of the present application, a video generating device for implementing the above embodiment is also provided. Fig.14 is a schematic diagram of a video generating device according to an embodiment of the present application, such as Fig.14 As shown, the device comprises:
[0251] An acquisition module 140 is used to acquire an original video of the user and image information of a first garment, wherein the first garment is the garment to be worn by the user;
[0252] A first generating module 142 is used to identify the second clothing in the original video and generate a temporally smooth local focus, wherein the temporally smooth local focus is used to represent the position information of the second clothing in different video frames of the original video, and the second clothing is the clothing that the user has worn;
[0253] The second generating module 144 is used to generate a target video based on the original video, the image information and the temporally smoothed local focus, wherein the user in the target video wears the first clothing.
[0254] It should be noted that the acquisition module 140, the first generation module 142 and the second generation module 144 correspond to steps S902 to S906 in the embodiment, and the modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the contents disclosed in the above embodiments. It should be noted that the above modules or units may be hardware components or software components stored in a memory and processed by one or more processors, and the above modules may also be part of a device and run in an AR / VR device provided in the embodiment.
[0255] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in the embodiments, as well as the application scenario and implementation process, but is not limited to the scheme provided in the embodiments.
[0256] According to an embodiment of the present application, a video generating device for implementing the above embodiment is also provided. Fig.15 is a schematic diagram of a video generating device according to an embodiment of the present application, such as Fig.15 As shown, the device comprises:
[0257] A first display module 150 is used to display the original video of the biological object and the image information of the object to be displayed on the presentation screen of the virtual reality VR device or the augmented reality AR device, wherein the object to be displayed is an object to be displayed on the biological object;
[0258] A first generating module 152 is used to identify a target object in the original video and generate a temporally smooth local focus, wherein the temporally smooth local focus is used to characterize position information of the target object in different video frames of the original video, and the target object is an object that has been displayed on the biological object;
[0259] A second generating module 154 is used to generate a target video based on the original video, the image information and the temporally smoothed local focus, wherein the object to be displayed is displayed on the biological object in the target video;
[0260] The second display module 156 is used to drive the VR device or AR device to display the target video.
[0261] It should be noted that the first display module 150, the first generation module 152, the second generation module 154 and the second display module 156 correspond to steps S1002 to S1008 in the embodiment, and the modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the contents disclosed in the above embodiments. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors, and the above modules can also be run in the AR / VR device provided in the embodiment as part of the device.
[0262] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in the embodiments, as well as the application scenario and implementation process, but is not limited to the scheme provided in the embodiments.
[0263] According to an embodiment of the present application, a video generating device for implementing the above embodiment is also provided. Fig.16 is a schematic diagram of a video generating device according to an embodiment of the present application, such as Fig.16 As shown, the device comprises:
[0264] An acquisition module 160, configured to acquire an original video of a biological object and image information of an object to be displayed by calling a first interface, wherein the first interface includes a first parameter, a parameter value of the first parameter includes the original video and image information, and the object to be displayed is an object to be displayed on the biological object;
[0265] A first generating module 162 is used to identify a target object in the original video and generate a temporally smooth local focus, wherein the temporally smooth local focus is used to characterize position information of the target object in different video frames of the original video, and the target object is an object that has been displayed on the biological object;
[0266] A second generating module 164 is used to generate a target video based on the original video, the image information and the temporally smoothed local focus, wherein the object to be displayed is displayed on the biological object in the target video;
[0267] The output module 166 is configured to output the target video by calling a second interface, wherein the second interface includes a second parameter, and a parameter value of the second parameter includes the target video.
[0268] It should be noted that the acquisition module 160, the first generation module 162, the second generation module 164 and the output module 166 correspond to steps S1102 to S1108 in the embodiment, and the modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors, and the above modules can also be run in the AR / VR device provided in the embodiment as part of the device.
[0269] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in the embodiments, as well as the application scenario and implementation process, but is not limited to the scheme provided in the embodiments.
[0270] An embodiment of the present application may provide a computer terminal, which may include an AR / VR device, a server, and a client. The AR / VR device may be any AR / VR device in an AR / VR device group.
[0271] Optionally, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of the computer network.
[0272] In this embodiment, the computer terminal can execute the program code in the method.
[0273] Optionally, Fig.17 is a structural block diagram of a computer terminal according to an embodiment of the present application. Fig.17 As shown, the computer terminal A may include: one or more (only one is shown in the figure) processors 102, a memory 104, a storage controller, and a peripheral interface, wherein the peripheral interface is connected to a radio frequency module, an audio module, and a display.
[0274] Among them, the memory can be used to store software programs and modules, such as program instructions / modules corresponding to the methods and devices in the embodiments of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, implementing the methods in the above embodiments. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer terminal A via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0275] The processor can call the information and application program stored in the memory through the transmission device to perform the following steps: obtain the original video of the biological object and the image information of the object to be displayed, wherein the object to be displayed is the object to be displayed on the biological object; identify the target object in the original video and generate a temporally smooth local focus, wherein the temporally smooth local focus is used to characterize the position information of the target object in different video frames of the original video, and the target object is the object currently loaded on the biological object as a carrier; use the temporally smooth local focus to amplify the original video to obtain an amplified video; input the amplified video into the video generation model, and guide the video generation model to generate a target video based on the temporally smooth local focus and the image information, wherein the object to be displayed is displayed on the biological object in the target video.
[0277] Optionally, the processor may also execute program codes of the following steps: respectively identifying target objects in multiple video frames of the original video to obtain initial display areas of the multiple video frames, wherein the initial display areas of different video frames are used to characterize position information of the target objects in different video frames; expanding the initial display areas of the multiple video frames to obtain target display areas of the multiple video frames; and a combining unit, used to combine the target display areas of the multiple video frames according to the time sequence of the multiple video frames to obtain a local focus with smooth temporal sequence.
[0278] Optionally, the processor may also execute the program code of the following steps: determining the posture information of the biological object in the original video; based on the posture information and the type of the object to be displayed, identifying the target object in multiple video frames of the original video respectively, and obtaining the initial display areas of the multiple video frames.
[0279] Optionally, the processor may also execute the program code of the following steps: determining the posture information of the biological object in the original video; based on the posture information and the type of the object to be displayed, identifying the target object in multiple video frames of the original video respectively, and obtaining the initial display area of the multiple video frames.
[0280] Optionally, the processor may also execute the program code of the following steps: combining the target display areas of the multiple video frames according to the time sequence of the multiple video frames to obtain an initial time-series smooth local focus; smoothing the initial time-series smooth local focus to obtain a time-series smooth local focus.
[0281] Optionally, the processor may also execute program codes of the following steps: based on the display area of the target object in the original video, processing the original video to obtain a first video, wherein the first video does not contain the target object; and amplifying the first video using time-smoothed local focusing to obtain an amplified video.
[0282] Optionally, the processor may also execute the following program code: extracting features from the temporally smoothed local focus to obtain temporally smoothed local focus features, extracting features from the magnified video to obtain object features, and extracting features from the image information to obtain object features; inputting the object features into a video generation model, and guiding the video generation model to generate a predicted video based on the temporally smoothed local focus features and the object features; and fusing the predicted video with the original video to obtain a target video.
[0283] Optionally, the processor may also execute the following program code: encoding the time-smoothed locally focused tunnel information to obtain a one-dimensional position code; and processing the one-dimensional position code using a linear mapping and an activation function to obtain a time-smoothed locally focused feature.
[0284] Optionally, the processor may also execute program codes of the following steps: obtaining a noise video corresponding to the enlarged video; determining a mask video corresponding to the display area; and performing feature extraction on the noise video, the mask video and the first video to obtain object features.
[0285] Optionally, the processor may also execute the program code of the following steps: using a posture encoder to perform feature extraction on the posture information of the biological object in the original video to obtain posture features; performing feature extraction on the noise video, the mask video and the first video respectively to obtain noise features corresponding to the noise video, mask features corresponding to the mask video, and target features corresponding to the first video; splicing the noise features, the mask features and the target features to obtain a first spliced features; adding the first spliced features to the posture features to obtain object features.
[0286] Optionally, the processor may also execute the program code of the following steps: extracting features from the image information using an image encoder to obtain image features of the image information; extracting features from the image information using an image diffusion model to obtain object texture features of the image information; wherein the object features include image features and object texture features.
[0287] Optionally, the processor may also execute the program code of the following steps: using an environment encoder to extract features from the first video to obtain environment features, wherein the environment features are used to characterize the content other than the target object in the original video; inputting the object features into a video generation model, and guiding the video generation model to generate a predicted video based on time-smoothed local focus features, environment features, and object features.
[0288] Optionally, the processor may also execute the program code of the following steps: using any spatial attention module to perform attention processing on input features, environmental features and object features to obtain a first attention feature output by any spatial attention module, wherein, when any attention module is the first module in the video generation model, the input feature is an object feature, and the first attention feature output by any attention module is output to a temporal attention module connected to any attention module; using any temporal attention module to perform attention processing on the input first attention feature and the temporally smoothed local focus feature to obtain a second attention feature output by any temporal attention module; reconstructing the second attention feature output by the last temporal attention module among multiple temporal attention modules to generate a predicted video.
[0289] Optionally, the processor may also execute the following program code: concatenating the input feature and the texture feature of the item in the item feature to obtain a second concatenated feature; performing self-attention processing on the second concatenated feature using a reference attention module to obtain a self-attention feature; concatenating the environment feature and the image feature in the item feature to obtain a third concatenated feature; and performing cross-attention processing on the third concatenated feature and the self-attention feature using the environment attention module to obtain a first attention feature.
[0290] Optionally, the processor may also execute the following program code: concatenating the temporally smoothed local focus feature and the first attention feature to obtain a fourth concatenated feature; and performing self-attention processing on the fourth concatenated feature using a self-attention module to obtain a second attention feature.
[0291] Optionally, the processor may also execute the program code of the following steps: using a Gaussian kernel to process the predicted video to obtain processed features; and fusing the processed features with the original video to obtain a target video.
[0292] Optionally, the processor may also execute program code for the following steps: obtaining first training data, wherein the first training data comprises a first training image displaying a first object and a second training image displaying a second object; using the first training data to train the video generation model, the posture encoder, and the image diffusion model; obtaining second training data, wherein the second training data comprises a first training video displaying the first object and a second training video displaying the second object; and using the second training data to train the video generation model and the environment encoder. The processor can call the information and application program stored in the memory through the transmission device to perform the following steps: in response to the image input instruction acting on the operation interface, the original video of the biological object and the image information of the object to be displayed are displayed on the operation interface, wherein the object to be displayed is an object to be displayed on the biological object; in response to the generation instruction acting on the operation interface, the target video is displayed on the biological object in the target video, and the target video is a video generated based on the original video, the image information and the temporally smoothed local focus, the temporally smoothed local focus is generated by identifying the target object in the original video, and the temporally smoothed local focus is used to characterize the position information of the target object in different video frames of the original video, and the target object is an object that has been displayed on the biological object.
[0293] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtain the user's original video and image information of the first clothing, wherein the first clothing is the clothing to be worn by the user; identify the second clothing in the original video and generate a time-smooth local focus, wherein the time-smooth local focus is used to characterize the position information of the second clothing in different video frames of the original video, and the second clothing is the clothing already worn by the user; based on the original video, the image information and the time-smooth local focus, generate a target video, wherein the user in the target video wears the first clothing.
[0294] The processor can call the information and application programs stored in the memory through the transmission device to perform the following steps: displaying the original video of the biological object and the image information of the object to be displayed on the presentation screen of the virtual reality VR device or the augmented reality AR device, wherein the object to be displayed is the object to be displayed on the biological object; identifying the target object in the original video and generating a temporally smooth local focus, wherein the temporally smooth local focus is used to characterize the position information of the target object in different video frames of the original video, and the target object is the object that has been displayed on the biological object; generating a target video based on the original video, the image information and the temporally smooth local focus, wherein the object to be displayed is displayed on the biological object in the target video; and driving the VR device or the AR device to display the target video.
[0295] The processor can call the information and application program stored in the memory through the transmission device to perform the following steps: obtain the original video of the biological object and the image information of the object to be displayed by calling the first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter includes the original video and the image information, and the object to be displayed is an object to be displayed on the biological object; identify the target object in the original video and generate a temporally smoothed local focus, wherein the temporally smoothed local focus is used to characterize the position information of the target object in different video frames of the original video, and the target object is an object that has been displayed on the biological object; generate a target video based on the original video, the image information and the temporally smoothed local focus, wherein the object to be displayed is displayed on the biological object in the target video; output the target video by calling the second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter includes the target video.
[0296] By adopting the embodiment of the present application, a scheme of a video generation method is provided. By acquiring the original video of the biological object and the image information of the object to be displayed; identifying the target object in the original video, generating a time-smooth local focus; using the time-smooth local focus to enlarge the original video, obtaining an enlarged video; inputting the enlarged video into the video generation model, and guiding the video generation model to generate the target video based on the time-smooth local focus and image information. It is easy to notice that the stable close-up of the area where the target object is located in the original video is extracted by the time-smooth local focus, so as to achieve the purpose of accurately locating the target object, and the original video is enlarged by the time-smooth local focus, so as to enlarge the area where the target object is located, better retain the details of the target object, and provide a more accurate reference for the placement and display details of the object to be displayed, thereby improving the presentation effect of the target video generated based on the enlarged video under the guidance of the time-smooth local focus and image information, thereby solving the technical problem of poor video effect presented by the video virtual try-on.
[0297] Those skilled in the art will appreciate that the structure shown in the figure is for illustration only, and the computer terminal A may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (MID), a PAD, and other terminal devices. The figure does not limit the structure of the above electronic devices. For example, the computer terminal A may also include more or fewer components (such as a network interface, a display device, etc.) than those shown in the figure, or have a configuration different from that shown in the figure.
[0298] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.
[0299] The embodiment of the present application further provides a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store the program code executed by the method provided in the above embodiment.
[0300] Optionally, in this embodiment, the computer-readable storage medium may be located in any one of the computer terminals in the computer terminal cluster, or in any one of the mobile terminals in the mobile terminal group.
[0301] Optionally, in this embodiment, the computer-readable storage medium is configured to store program codes for executing the following steps: obtaining an original video of a biological object and image information of an object to be displayed, wherein the object to be displayed is an object to be displayed on the biological object; identifying a target object in the original video and generating a temporally smooth local focus, wherein the temporally smooth local focus is used to characterize position information of the target object in different video frames of the original video, and the target object is an object currently loaded on the biological object as a carrier; amplifying the original video using the temporally smooth local focus to obtain an amplified video; inputting the amplified video into a video generation model, and guiding the video generation model to generate a target video based on the temporally smooth local focus and image information, wherein the object to be displayed is displayed on the biological object in the target video.
[0302] Optionally, the computer readable storage medium is further configured to store program codes for performing the following steps:
[0303] Optionally, target objects in multiple video frames of the original video are identified respectively to obtain initial display areas of the multiple video frames, wherein the initial display areas of different video frames are used to characterize position information of the target object in different video frames; the initial display areas of the multiple video frames are expanded to obtain target display areas of the multiple video frames; a combining unit is used to combine the target display areas of the multiple video frames according to the time sequence of the multiple video frames to obtain a local focus with smooth temporal sequence.
[0304] Optionally, the posture information of the biological object in the original video is determined; based on the posture information and the type of the object to be displayed, the target object in multiple video frames of the original video is identified respectively to obtain the initial display areas of the multiple video frames.
[0305] Optionally, determine the posture information of the biological object in the original video; based on the posture information and the type of the object to be displayed, identify the target object in multiple video frames of the original video respectively to obtain the initial display areas of the multiple video frames.
[0306] Optionally, according to the time sequence of the multiple video frames, the target display areas of the multiple video frames are combined to obtain a local focus with initial time sequence smoothness; and the local focus with initial time sequence smoothness is smoothed to obtain a local focus with time sequence smoothness.
[0307] Optionally, based on the display area of the target object in the original video, the original video is processed to obtain a first video, wherein the first video does not contain the target object; the first video is enlarged using time-smoothed local focusing to obtain an enlarged video.
[0308] Optionally, feature extraction is performed on the temporally smoothed local focus to obtain temporally smoothed local focus features, feature extraction is performed on the magnified video to obtain object features, and feature extraction is performed on the image information to obtain item features; the object features are input into a video generation model, and the video generation model is guided to generate a predicted video based on the temporally smoothed local focus features and the item features; the predicted video is fused with the original video to obtain a target video.
[0309] Optionally, the time-smoothed locally focused tunnel information is encoded to obtain a one-dimensional position code; the one-dimensional position code is processed using a linear mapping and an activation function to obtain a time-smoothed locally focused feature.
[0310] Optionally, a noise video corresponding to the enlarged video is obtained; a mask video corresponding to the display area is determined; and features are extracted from the noise video, the mask video and the first video to obtain object features.
[0311] Optionally, a posture encoder is used to extract features of posture information of biological objects in the original video to obtain posture features; feature extraction is performed on the noise video, the mask video and the first video respectively to obtain noise features corresponding to the noise video, mask features corresponding to the mask video, and target features corresponding to the first video; the noise features, mask features and target features are spliced to obtain a first spliced feature; the first spliced feature is added to the posture feature to obtain an object feature.
[0312] Optionally, an image encoder is used to extract features from the image information to obtain image features of the image information; an image diffusion model is used to extract features from the image information to obtain object texture features of the image information; wherein the object features include image features and object texture features.
[0313] Optionally, an environment encoder is used to extract features from the first video to obtain environment features, wherein the environment features are used to characterize the content in the original video other than the target object; the object features are input into a video generation model, and the video generation model is guided to generate a predicted video based on time-smoothed local focus features, environment features, and object features.
[0314] Optionally, use any spatial attention module to perform attention processing on input features, environmental features and object features to obtain a first attention feature output by any spatial attention module, wherein, when any attention module is the first module in the video generation model, the input feature is an object feature, and the first attention feature output by any attention module is output to a temporal attention module connected to any attention module; use any temporal attention module to perform attention processing on the input first attention feature and the temporally smoothed local focus feature to obtain a second attention feature output by any temporal attention module; reconstruct the second attention feature output by the last temporal attention module among multiple temporal attention modules to generate a predicted video.
[0315] Optionally, the input features and the texture features of the item in the item features are spliced to obtain a second spliced feature; the second spliced feature is self-attention processed using the reference attention module to obtain a self-attention feature; the environment features and the image features in the item features are spliced to obtain a third spliced feature; the third spliced feature and the self-attention feature are cross-attention processed using the environment attention module to obtain the first attention feature.
[0316] Optionally, the temporally smoothed local focus feature and the first attention feature are spliced to obtain a fourth spliced feature; and the self-attention module is used to perform self-attention processing on the fourth spliced feature to obtain a second attention feature.
[0317] Optionally, the predicted video is processed using a Gaussian kernel to obtain processed features; the processed features are fused with the original video to obtain a target video.
[0318] Optionally, first training data is obtained, wherein the first training data includes a first training image showing a first object and a second training image showing a second object; the video generation model, the posture encoder, and the image diffusion model are trained using the first training data; second training data is obtained, wherein the second training data includes a first training video showing the first object and a second training video showing the second object; and the video generation model and the environment encoder are trained using the second training data.
[0319] Optionally, the computer-readable storage medium is also configured to store program codes for executing the following steps: in response to an image input instruction acting on an operation interface, displaying an original video of a biological object and image information of an object to be displayed on the operation interface, wherein the object to be displayed is an object to be displayed on the biological object; in response to a generation instruction acting on the operation interface, displaying a target video on the operation interface, wherein the object to be displayed is displayed on the biological object in the target video, and the target video is a video generated based on the original video, the image information, and a time-smoothed local focus, wherein the time-smoothed local focus is generated by identifying a target object in the original video, and the time-smoothed local focus is used to characterize the position information of the target object in different video frames of the original video, and the target object is an object that has been displayed on the biological object.
[0320] Optionally, the computer-readable storage medium is also configured to store program codes for executing the following steps: obtaining an original video of a user and image information of a first article of clothing, wherein the first article of clothing is the article of clothing to be worn by the user; identifying a second article of clothing in the original video and generating a temporally smooth local focus, wherein the temporally smooth local focus is used to characterize position information of the second article of clothing in different video frames of the original video, and the second article of clothing is the article of clothing already worn by the user; generating a target video based on the original video, the image information and the temporally smooth local focus, wherein the user in the target video is wearing the first article of clothing.
[0321] Optionally, the computer-readable storage medium is also configured to store program codes for executing the following steps: displaying an original video of a biological object and image information of an item to be displayed on a presentation screen of a virtual reality (VR) device or an augmented reality (AR) device, wherein the item to be displayed is an object to be displayed on the biological object; identifying a target object in the original video and generating a temporally smooth local focus, wherein the temporally smooth local focus is used to characterize position information of the target object in different video frames of the original video, and the target object is an object that has been displayed on the biological object; generating a target video based on the original video, the image information and the temporally smooth local focus, wherein the item to be displayed is displayed on the biological object in the target video; and driving the VR device or the AR device to display the target video.
[0322] Optionally, the computer-readable storage medium is also configured to store program code for executing the following steps: obtaining an original video of a biological object and image information of an object to be displayed by calling a first interface, wherein the first interface includes a first parameter, and a parameter value of the first parameter includes the original video and image information, and the object to be displayed is an object to be displayed on the biological object; identifying a target object in the original video and generating a temporally smooth local focus, wherein the temporally smooth local focus is used to characterize position information of the target object in different video frames of the original video, and the target object is an object that has been displayed on the biological object; generating a target video based on the original video, image information, and temporally smooth local focus, wherein the object to be displayed is displayed on the biological object in the target video; outputting the target video by calling a second interface, wherein the second interface includes a second parameter, and a parameter value of the second parameter includes the target video.
[0323] The embodiment of the present application further provides a computer program product. Optionally, in this embodiment, the computer program product may include a computer program, and the computer program implements the method provided in the embodiment when executed by a processor.
[0324] The embodiments of the present application also provide a computer program product. Optionally, the computer program product may include a non-volatile computer-readable storage medium, which may be used to store a computer program, and when the computer program is executed by a processor, the method provided in the embodiments is implemented.
[0325] The embodiment of the present application further provides a computer program. Optionally, in this embodiment, when the computer program is executed by a processor, the method provided in the above embodiment is implemented.
[0326] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.
[0327] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.
[0328] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.
[0329] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0330] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0331] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.
[0332] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.
Claims
1. A video generation method, characterized in that: include: Acquire an original video of a biological object and image information of an object to be displayed, wherein the object to be displayed is an object to be displayed on the biological object; Identify the target object in the original video and generate a temporally smoothed local focus, wherein the temporally smoothed local focus is used to characterize the position information of the target object in different video frames of the original video, and the target object is an object currently loaded on the biological object as a carrier; Amplifying the original video using the temporally smoothed local focus to obtain an amplified video; The enlarged video is input into a video generation model, and the video generation model is guided to generate a target video based on the temporally smoothed local focus and the image information, wherein the object to be displayed is displayed on the biological object in the target video.
2. The method according to claim 1, characterized in that The identifying the target object in the original video and generating a temporally smooth local focus includes: Respectively identifying the target object in multiple video frames of the original video to obtain initial display areas of the multiple video frames, wherein the initial display areas of different video frames are used to represent the position information of the target object in the different video frames; Expanding the initial display areas of the multiple video frames to obtain target display areas of the multiple video frames; According to the time sequence of the multiple video frames, the target display areas of the multiple video frames are combined to obtain the local focus with smooth time sequence.
3. The method according to claim 2, characterized in that The step of respectively identifying the target object in the multiple video frames of the original video to obtain the initial display areas of the multiple video frames includes: Determining posture information of the biological object in the original video; Based on the posture information and the type of the object to be displayed, the target object in the multiple video frames of the original video is identified respectively to obtain initial display areas of the multiple video frames.
4. The method according to claim 2, characterized in that: The step of combining the target display areas of the multiple video frames according to the time sequence of the multiple video frames to obtain the local focus with smooth temporal sequence includes: Combining the target display areas of the multiple video frames according to the time sequence of the multiple video frames to obtain a local focus with initial time sequence smoothness; The initial time-series smoothed local focus is smoothed to obtain the time-series smoothed local focus.
5. The method according to claim 1, characterized in that The method of amplifying the original video by using the temporally smoothed local focusing to obtain the amplified video includes: Based on the display area of the target object in the original video, the original video is processed to obtain a first video, wherein the first video does not contain the target object; The first video is amplified by using the temporally smoothed local focus to obtain the amplified video.
6. The method according to any one of claims 1 to 5, characterized in that The step of inputting the enlarged video into a video generation model, and guiding the video generation model to generate a target video based on the temporally smoothed local focus and the image information, comprises: Performing feature extraction on the temporally smoothed local focus to obtain temporally smoothed local focus features, performing feature extraction on the magnified video to obtain object features, and performing feature extraction on the image information to obtain item features; Inputting the object features into the video generation model, and guiding the video generation model to generate a predicted video based on the temporally smoothed local focus features and the object features; The predicted video is fused with the original video to obtain the target video.
7. The method according to claim 6, characterized in that The extracting features of the local focus of the time series smoothing to obtain the local focus features of the time series smoothing includes: Encoding the time-series smoothed locally focused tunnel information to obtain a one-dimensional position code; The one-dimensional position encoding is processed by using linear mapping and activation function to obtain the temporally smoothed local focus feature.
8. The method according to claim 6, characterized in that The step of extracting features from the enlarged video to obtain object features includes: Obtaining a noise video corresponding to the amplified video; Determine a mask video corresponding to the display area; Feature extraction is performed on the noise video, the mask video, and the first video to obtain the object features.
9. The method according to claim 8, characterized in that The extracting features of the noise video, the mask video and the first video to obtain the object features includes: Using a posture encoder to extract features from the posture information of the biological object in the original video to obtain posture features; Performing feature extraction on the noise video, the mask video, and the first video respectively to obtain noise features corresponding to the noise video, mask features corresponding to the mask video, and target features corresponding to the first video; Splicing the noise feature, the mask feature and the target feature to obtain a first splicing feature; The first splicing feature and the posture feature are added to obtain the object feature.
10. The method according to claim 6, characterized in that The step of extracting features from the image information to obtain object features includes: Using an image encoder to extract features from the image information to obtain image features of the image information; Extracting features from the image information using an image diffusion model to obtain texture features of the object in the image information; The object features include the image features and the object texture features.
11. The method according to claim 6, characterized in that The step of inputting the object features into a video generation model, and guiding the video generation model to generate a predicted video based on the temporally smoothed local focus features and the object features, comprises: Using an environment encoder to extract features from the first video to obtain environment features, wherein the environment features are used to characterize content other than the target object in the original video; The object features are input into a video generation model, and the video generation model is guided to generate the predicted video based on the temporally smoothed local focus features, the environment features, and the object features.
12. The method according to claim 11, characterized in that The video generation model comprises: a plurality of spatial attention modules and a plurality of temporal attention modules connected alternately; the step of inputting the object features into the video generation model, and guiding the video generation model to generate the predicted video based on the temporally smoothed local focus features, the environmental features and the object features, comprises: Using any one of the spatial attention modules to perform attention processing on the input feature, the environmental feature, and the object feature, to obtain a first attention feature output by the any one of the spatial attention modules, wherein, when the any one of the attention modules is the first module in the video generation model, the input feature is the object feature, and the first attention feature output by the any one of the attention modules is output to a temporal attention module connected to the any one of the attention modules; Using any one of the time attention modules to perform attention processing on the input first attention feature and the temporally smoothed local focus feature, to obtain a second attention feature output by the any one of the time attention modules; The second attention feature output by the last temporal attention module among the multiple temporal attention modules is reconstructed to generate the predicted video.
13. The method according to claim 12, characterized in that Any one of the spatial attention modules includes: a reference attention module and an environment attention module; using any one of the spatial attention modules to perform attention processing on the input feature, the environment feature and the object feature to obtain a first attention feature output by any one of the spatial attention modules includes: splicing the input feature and the object texture feature in the object feature to obtain a second splicing feature; Using the reference attention module to perform self-attention processing on the second splicing feature to obtain a self-attention feature; splicing the environment feature and the image feature in the object feature to obtain a third splicing feature; The environmental attention module is used to perform cross-attention processing on the third splicing feature and the self-attention feature to obtain the first attention feature.
14. The method according to claim 12, characterized in that Any one of the temporal attention modules comprises: a self-attention module; the using any one of the temporal attention modules to perform attention processing on the input first attention feature and the temporally smoothed local focus feature to obtain a second attention feature output by any one of the temporal attention modules, comprises: Concatenate the temporally smoothed local focus feature and the first attention feature to obtain a fourth concatenated feature; The self-attention module is used to perform self-attention processing on the fourth splicing feature to obtain the second attention feature.
15. The method according to claim 6, characterized in that The step of fusing the predicted video with the original video to obtain the target video includes: Processing the predicted video using a Gaussian kernel to obtain processed features; The processed features are fused with the original video to obtain the target video.
16. The method according to any one of claims 1 to 15, characterized in that The method further comprises: Acquire first training data, wherein the first training data includes a first training image showing a first object and a second training image showing a second object; Using the first training data to train the video generation model, the gesture encoder, and the image diffusion model; Acquire second training data, wherein the second training data includes a first training video showing the first object and a second training video showing the second object; The video generation model and the environment encoder are trained using the second training data.
17. A video generation method, characterized in that: include: In response to an image input instruction acting on the operation interface, an original video of the biological object and image information of an object to be displayed are displayed on the operation interface, wherein the object to be displayed is an object to be displayed on the biological object; In response to a generation instruction acting on the operation interface, a target video is displayed on the operation interface, wherein the object to be displayed is displayed on the biological object in the target video, and the target video is a video generated based on the original video, the image information and temporally smoothed local focus, the temporally smoothed local focus is generated by identifying the target object in the original video, and the temporally smoothed local focus is used to characterize the position information of the target object in different video frames of the original video, and the target object is an object that has been displayed on the biological object.
18. A video generation method, characterized in that: include: Displaying the original video of the biological object and the image information of the object to be displayed on the presentation screen of the virtual reality (VR) device or the augmented reality (AR) device, wherein the object to be displayed is an object to be displayed on the biological object; Identify the target object in the original video and generate a temporally smooth local focus, wherein the temporally smooth local focus is used to characterize the position information of the target object in different video frames of the original video, and the target object is an object displayed on the biological object; Generate a target video based on the original video, the image information and the temporally smoothed local focus, wherein the object to be displayed is displayed on the biological object in the target video; The VR device or the AR device is driven to display the target video.
19. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored executable program, wherein when the executable program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 18.
20. A computer program product, characterized in that The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 18.
Citation Information
Cited By
Video generation method and device based on clothing image, storage medium and product
CN120529030A
Video segmentation method and system
CN120913134A
A video segmentation method and system
CN120913134B
