Track guide video generation method and device based on entity and position relation

Abstract

The invention provides a track guide video generation method and device based on entity and position relation, and the method comprises the steps: obtaining a to-be-edited video frame and a corresponding motion track, and the video frame comprises a plurality of entities; an extraction step: establishing a polar coordinate system by taking each entity as a center, converting four-dimensional vectors of bounding boxes of adjacent entities into polar coordinate vectors to obtain relative positions between the entities, and combining the relative positions and entity semantic information of the video frames to obtain image features of the video frames; a zero matrix is initialized, image features are embedded according to track sequence points of the motion track, and entity representation is obtained; and inputting the entity representation and the motion trail into an encoder to obtain an encoding result, superposing the encoding result and Gaussian noise, inputting the superposed encoding result and Gaussian noise into a de-noising diffusion model to obtain a plurality of features with different resolutions as potential condition signals, and finally decoding the potential condition signals through a decoder to obtain a reconstructed video. And taking the result as a video generation result of the video frame under the motion trail.

Description

Technical Field

[0001] The method belongs to the field of video generation technology for computer vision in the field of machine learning and computer vision, and particularly relates to a method and device for generating a trajectory-guided video based on entity and position relationship. Background Art

[0002] In recent years, in the field of video generation, trajectory-based control generation methods have attracted more and more researchers' attention due to their good user interaction characteristics. The existing technology first predicts the optical flow map by encoding images and trajectories. However, simply distorting the features with the help of optical flow will cause unnatural distortion of the object. In order to solve this problem, the existing technology later proposed to compress the video into a dense latent space, and use the RNN network to gradually decode these latent variables, and predict future frames in an autoregressive manner through the diffusion model. Although it shows good results, it relies heavily on more than a dozen prior key points extracted through the HRNet network, so it can only control human body movements, and the generated video cannot move along the predetermined trajectory.

[0003] Encode sparse trajectories into dense optical flow sequences, and then use these sequences as supervisory signals to control the movement of objects. Or directly encode the trajectory coordinates of each object into a vector field, and use this vector field as a condition to control the movement of objects. However, there are still some problems with the above research. First, a single pixel cannot represent an object, and dragging a single pixel cannot accurately control the corresponding object. Secondly, when processing human datasets such as TED-Talk and Human 3.6M, multiple complex trajectories make the pixels close to the trajectory move faster, resulting in serious distortion of the human body.

[0004] Based on the above observations, this task needs to solve two problems: 1) It is necessary to determine what is an entity in the picture, that is, to identify the specific area to be dragged; 2) How to drag, that is, how to achieve accurate dragging of the selected area. For the first challenge, although SAM can be used in the initial frame to conveniently select the area that the present invention wants to control. However, for the distortion of individual humans, more refined segmentation is still required, such as limbs and heads. For the second challenge, the present invention proposes a new entity representation method that integrates the spatial relationship between objects to achieve precise motion control of entities in videos and generate action videos. Summary of the invention

[0005] The present invention provides a method for generating a trajectory-guided video based on entity and positional relationship. The present invention provides a method for generating a trajectory-guided video based on entity and positional relationship. Specifically, the present invention uses SAM as a basic segmentation model, uses LIP to perform fine segmentation on the human body, and obtains an entity mask of the first frame. Then, using the mask of each entity in the first frame, the center coordinates of the entity are extracted to generate an entity representation, and the relative spatial position relationship is fused in the entity representation through a positional relationship perception module. Finally, according to the input motion trajectory points, the motion trajectory of the center coordinates predicted by the collaborative tracker CoTrack is used as the motion trajectory of the entire entity.

[0006] Specifically, in view of the shortcomings of existing technologies, such as Figure 3 As shown, the present invention proposes a trajectory guided video generation method based on entity and position relationship, which includes:

[0007] The initial step is to obtain a video frame to be edited and its corresponding motion trajectory, wherein the video frame includes multiple entities; and construct an SVD model including a denoising diffusion model, an encoder and a decoder;

[0008] In the extraction step, a polar coordinate system is established with each entity as the center, and the relative positions between entities are obtained by converting the four-dimensional vectors of the bounding boxes of adjacent entities into polar coordinate vectors. The image features of the video frame are obtained by combining the relative positions with the entity semantic information of the video frame;

[0009] In a representation step, a zero matrix corresponding to the size of the video frame is initialized, and the image features are embedded according to the trajectory sequence points of the motion trajectory to obtain an entity representation;

[0010] In the reconstruction step, the entity representation and the motion trajectory are input into the encoder to obtain an encoding result, the encoding result is superimposed with Gaussian noise and then input into the denoising diffusion model to obtain multiple features of different resolutions as potential conditional signals, and finally the potential conditional signal is decoded by the decoder to obtain a reconstructed video as the video generation result of the video frame under the motion trajectory.

[0011] The trajectory-guided video generation method based on entity and position relationship, wherein the trajectory sequence points of the motion trajectory are , ,in Indicates the length of the video.

[0012] The method for generating trajectory-guided videos based on entity and position relationship, wherein the extraction step comprises:

[0013] For each entity ,by Establish a polar coordinate system for the center and place the adjacent objects of i The 4D vector of the bounding box Convert to polar coordinates ;

[0014] The low-dimensional spatial position vector is embedded into a higher-dimensional space through a set of Gaussian kernels with learnable mean and covariance, and the entity i and The spatial relationship dimension after embedding ;entity by The Gaussian kernel operation centered on is defined as follows:

[0015] )

[0016] )

[0017] in, and is the Gaussian distribution of relative distances with learnable mean and covariance, and is the Gaussian distribution of angles with learnable mean and covariance;

[0018] Aggregate entities by a scaling function and The relative distance and angle relationship of the image, the aggregated spatial weight of the image is expressed as :

[0019]

[0020] in, is the number of entities in the video frame;

[0021] Image features pass Perform a linear transformation, is the dimension of the image feature characteristics;

[0022]

[0023] The relative position relationship between entities in the image is fused with the semantic relationship and scaled to the range of (0, 1) by the following formula:

[0024]

[0025] The multi-head attention mechanism is used to focus on the information in the input sequence from different angles, capturing richer image feature information;

[0026]

[0027] Where k is the number of attention heads.

[0028] By image features , obtain the corresponding entity representation from the index coordinates in the segmentation mask of the video frame; get the final image feature , where k represents the number of entities and the channel size is ;

[0029] The representation step includes: initializing a zero matrix , then embed the entity vector according to the trajectory sequence points, and use the entity mask of the video frame to extract the center coordinates of the entity , As the starting point of each trajectory sequence point, the entity representation is embedded into the corresponding zero matrix through the starting point coordinate index In order to obtain the final entity representation .

[0030] The present invention also proposes a trajectory guidance video generation device based on entity and position relationship, which includes:

[0031] The initial module obtains the video frame to be edited and its corresponding motion trajectory, wherein the video frame includes multiple entities; constructs an SVD model including a denoising diffusion model, an encoder and a decoder;

[0032] The extraction module establishes a polar coordinate system with each entity as the center, obtains the relative positions of the entities by converting the four-dimensional vectors of the bounding boxes of adjacent entities into polar coordinate vectors, and obtains the image features of the video frame by combining the relative positions with the entity semantic information of the video frame;

[0033] The representation module initializes a zero matrix corresponding to the size of the video frame, embeds the image features according to the trajectory sequence points of the motion trajectory, and obtains the entity representation;

[0034] The reconstruction module inputs the entity representation and the motion trajectory into the encoder to obtain the encoding result, superimposes the encoding result with Gaussian noise and inputs it into the denoising diffusion model to obtain multiple features of different resolutions as potential conditional signals, and finally decodes the potential conditional signal through the decoder to obtain the reconstructed video as the video generation result of the video frame under the motion trajectory.

[0035] The trajectory-guided video generation device based on entity and position relationship, wherein the trajectory sequence points of the motion trajectory are , ,in Indicates the length of the video.

[0036] The trajectory guidance video generation device based on entity and position relationship, wherein the extraction module comprises:

[0037] For each entity ,by Establish a polar coordinate system for the center and place the adjacent objects of i The 4D vector of the bounding box Convert to polar coordinates ;

[0038] The low-dimensional spatial position vector is embedded into a higher-dimensional space through a set of Gaussian kernels with learnable mean and covariance, and the entity i and The spatial relationship dimension after embedding ;entity by The Gaussian kernel operation centered on is defined as follows:

[0039] )

[0040] )

[0041] in, and is the Gaussian distribution of relative distances with learnable mean and covariance, and is the Gaussian distribution of angles with learnable mean and covariance;

[0042] Aggregate entities by a scaling function and The relative distance and angle relationship of the image, the aggregated spatial weight of the image is expressed as :

[0043]

[0044] in, is the number of entities in the video frame;

[0045] Image features pass Perform a linear transformation, is the dimension of the image feature characteristics;

[0046]

[0047] The relative position relationship between entities in the image is fused with the semantic relationship and scaled to the range of (0, 1) by the following formula:

[0048]

[0049] The multi-head attention mechanism is used to focus on the information in the input sequence from different angles, capturing richer image feature information;

[0050]

[0051] Where k is the number of attention heads.

[0052] By image features , obtain the corresponding entity representation from the index coordinates in the segmentation mask of the video frame; get the final image feature , where k represents the number of entities and the channel size is ;

[0053] The representation module includes: initializing a zero matrix , then embed the entity vector according to the trajectory sequence points, and use the entity mask of the video frame to extract the center coordinates of the entity , As the starting point of each trajectory sequence point, the entity representation is embedded into the corresponding zero matrix through the starting point coordinate index In order to obtain the final entity representation .

[0054] The present invention also proposes an electronic device, which includes the above-mentioned trajectory guidance video generation device based on entity and position relationship.

[0055] The electronic device is connected to an information display device, and the information display device is used to display the video generation result according to display parameters and attributes set by the user or through an artificial intelligence model.

[0056] The present invention also proposes a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for generating a trajectory-guided video based on entity and position relationship are implemented.

[0057] The present invention also proposes a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the method for generating a trajectory-guided video based on entity and position relationship are implemented.

[0058] It can be seen from the above scheme that the advantages of the present invention are:

[0059] The present invention mainly solves the problems of distortion and spatial position disorder that occur when complex motion trajectories drive multiple objects in an image. The method uses entity and position relationship information to guide the trajectory, ensuring the structural integrity and spatial consistency of the object during the trajectory dragging process. Specifically, the method mainly consists of three parts: a denoising diffusion model is used to learn the denoising process in spatial and temporal dimensions; the encoder is responsible for encoding the supervisory signal into the latent vector space; and the decoder is responsible for reconstructing the image from the latent representation. Unlike previous work, this method designs an entity representation mechanism in the encoder. The mechanism first uses a segmentation model to obtain the entity mask of the first frame, and then extracts the center coordinates of each entity based on the mask of the first frame to generate an entity representation, and fuses the relative spatial position relationship information in the entity representation through a position relationship perception module. This method allows multiple trajectories to control multiple objects in an image at the same time, improving the fine-grained control capability of video generation.

[0060] The experimental results demonstrate the superior performance of the trajectory-guided video generation method based on entity and position relationship proposed in the present invention in terms of entity integrity and trajectory consistency during the object dragging process, verifying the effectiveness of the method.

[0061] The present invention is tested on the Pytorch platform, and all models are trained with 8 GTX A100 GPUs. All training is based on the stable video diffusion (SVD) architecture and weights, which can generate 20 frames with a resolution of 320 × 576. The present invention uses Adam as the optimizer, and the learning rate is set to 1e-4, which is decayed every 10 rounds, with a decay factor of 0.1 each time and a minimum of 1e-6.

[0062] Experimental results:

[0063] In order to verify the effectiveness of the present invention, VIPSeg, WebVid and TED-talks are used as the test sets of the present invention in the experiment. Since FVD requires videos to have the same resolution and duration, the present invention adjusts the VIPSeg verification dataset to 256×256 resolution and reduces its length to 20 frames for evaluation. The present invention uses VIPSeg and WebVid as training sets and adopts a collaborative tracker to obtain the corresponding motion trajectories.

[0064] The present invention adopts two evaluation indicators. (1) Video quality evaluation: The present invention uses Frechette perceptual distance (FID) and Frechette video distance (FVD) to evaluate visual quality and temporal consistency. (2) Object motion control performance evaluation: The Euclidean distance (ObjMC) between the predicted object trajectory and the true trajectory is used to evaluate the object motion control performance. In addition, in the user study, the present invention randomly generated 50 videos and asked 50 users to vote on the quality and motion trajectory of the generated videos.

[0065] The present invention first experiments this method on the WebVid dataset and compares it with the existing methods. The experimental results are shown in Table 1:

[0066]

[0067] Table 1 Performance comparison on the WebVid dataset

[0068] Table 1 shows the video quality comparison using the FID metric on the WebVid dataset. The present invention compares the performance between the method of the present invention and DragNUWA, with other conditions being the same (basic architecture). The FID score of the present invention reached 34.5, which is significantly better than the current SOTA model DragNUWA (34.5 vs. 36.9). According to MotionCtrl, ObjMC is used to evaluate the motion control performance by calculating the Euclidean distance between the predicted trajectory and the true trajectory. Compared with DragNUWA, the model of the present invention achieved the best performance with a score of 302.7. In addition, FVD evaluates the temporal consistency of the generated video by comparing the feature distribution between the generated video and the real video. Compared with the performance of DragNUWA (521.7), the model of the present invention also performed better in temporal consistency, with a score of 510.8, a significant improvement of 10.9.

[0069]

[0070] Table 2 Ablation study of entity representation and spatial position relationship

[0071] Entity representation and spatial relations are the core components of the present work. The present invention keeps other conditions unchanged and only modifies the corresponding conditional features. Table 2 shows the ablation study of these two aspects. Since the entity representation mainly affects the object motion in the generated video, the present invention only needs to compare ObjMC, while the FVD and FID indicators only focus on temporal consistency and overall video quality. With entity representation, the ObjMC of the model saw a significant improvement to 302.7.

[0072]

[0073] Table 3. User evaluation results.

[0074] User Evaluation We conducted a user survey to evaluate the video realism from MotionCtrl, DragNVWA, and our method, each with 576×320 resolution. Ten volunteers selected the best method in terms of: 1) structural integrity; 2) trajectory consistency; 3) overall quality. Table 3 shows that our method outperforms other methods in terms of details, trajectory motion, and image quality.

[0075] The present invention is visualized and compared on the TedTalk dataset. It can be observed that when multiple trajectories act on the human body at the same time, the DragNVWA model exhibits phenomena such as feature distortion, artifacts, and multiple hands. The model of the present invention can accurately move according to the motion trajectory while maintaining the basic structure of the human body, thus proving the effectiveness of the method of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 Schematic diagram of trajectory-guided video generation method based on entity and position relationship;

[0077] Figure 2 Schematic diagram of the position relationship perception module;

[0078] Figure 3 is a flow chart of the method of the present invention;

[0079] Figure 4 It is a module diagram of the device of the present invention;

[0080] Figure 5 This is a schematic diagram of the structure of a first electronic device of the present invention;

[0081] Figure 6 This is a schematic diagram of the application environment structure of the first electronic device of the present invention;

[0082] Figure 7 It is a schematic diagram of the structure of a second electronic device of the present invention.

[0083] Reference numerals:

[0084] A-First electronic device;

[0085] B-Trajectory-guided video generation device based on entity and position relationship;

[0086] C-data acquisition equipment;

[0087] D-information display device;

[0088] 1000 - second electronic device;

[0089] Ⅰ-computational unit;

[0090] II-ROM;

[0091] III-RAM;

[0092] IV-bus;

[0093] V-interface;

[0094] VI - input unit;

[0095] VII-output unit;

[0096] VIII- Storage medium;

[0097] Ⅸ-Communication unit. DETAILED DESCRIPTION

[0098] It should be noted that, in this application, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0099] Without more constraints, an element defined by the phrase "comprising a..." does not exclude the existence of other identical elements in the process, method, article or apparatus comprising the element.

[0100] The processor described in the present invention is the control center of the electronic device, which can be a processor or a general term for multiple processing elements. For example, it can be one or more central processing units (CPUs), or application specific integrated circuits (ASICs), or one or more integrated circuits configured to implement the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (FPGAs).

[0101] Optionally, the processor can perform various functions of the electronic device by running or executing a software program stored in the memory, and calling data stored in the memory.

[0102] In a specific implementation, as an embodiment, the processor may include one or more CPUs. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions). Electronic devices may include: servers, desktop computers, laptops, smart phones, tablet computers, embedded computers, etc., wherein the embedded computers include vehicles and robots, etc.

[0103] The memory is used to store the software program for executing the solution of the present invention, and the execution is controlled by the processor. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0104] It should be noted that the structure of the electronic device shown in the drawings of the present invention does not constitute a limitation thereto, and the actual knowledge structure recognition device may include more or fewer components than shown in the drawings, or a combination of certain components, or a different arrangement of components.

[0105] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0106] It should also be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0107] In the present invention, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0108] It should also be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0109] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0110] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0111] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0112] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0113] The method for generating trajectory guidance videos based on entity and position relationship proposed in this invention uses entity representation and position relationship information to guide the trajectory, thus ensuring the structural integrity and spatial consistency of the object during the trajectory dragging process. The network contains the following key technical points:

[0114] (1) The present invention transforms the basic unit of trajectory control from the pixel point of the image into a single entity or object, proposes an entity representation mechanism in the encoder, and introduces entity mask prior knowledge to achieve true entity-level motion feature representation.

[0115] (2) A position relationship perception module is proposed. This module models the relative positions between object bounding boxes, such as up and down, left and right, through a cross-attention mechanism, thereby integrating the relative spatial position relationship information into the entity feature representation.

[0116] (3) In the training phase, the present invention constructs motion trajectories of entities with relative position relationships. First, for each entity, its center coordinates and radius are obtained using its corresponding mask. Then, the motion trajectory of its center is determined by the collaborative tracker as the representative motion trajectory of the entity to eliminate the influence of inconsistent trajectory motion during the dragging process of different pixel points within a single entity.

[0117] (4) The algorithm proposed in this invention has good controllability in video generation. When multiple trajectories act on the same object, the movement of each part must maintain a relative spatial relationship to maintain the structural integrity of the object. This invention uses latent features to represent each entity and embeds the spatial relative position relationship into the entity representation to ensure that the movement of the entity in the image can be controlled while maintaining the structural integrity.

[0118] In order to make the above features and effects of the present invention more clearly and understandably described, embodiments are given below and described in detail with reference to the accompanying drawings. This specification discloses one or more embodiments that include the features of the present invention. The disclosed embodiments are only for illustration. The scope of protection of the present invention is not limited to the disclosed embodiments, and the present invention is defined by the attached claims.

[0119] The present invention starts from the general situation of trajectory-guided video generation and explains the trajectory-guided video generation method based on the entity (target object) and position relationship proposed by the present invention.

[0120] The task of trajectory-based controllable video generation is to generate a video based on a given image and motion trajectory. Given a point trajectory , ,in Represents the length of the video, using a conditional denoising autoencoder To generate a video corresponding to the motion trajectory, control signal Contains two types of information: trajectory points and the first frame of the video, that is, the initial input image.

[0121] The overall process of the present invention is as follows Figure 1 As shown, the singular value decomposition SVD model is used as the basic architecture of the present invention, which mainly includes three parts: a denoising diffusion model (3D U-Net), which learns the denoising process in spatial and temporal dimensions; an encoder and a decoder, which encode the supervision signal into the latent space and reconstruct the denoised latent features back to the video. The present invention uses 3D U-Net to encode the guidance signal. The guidance signal of the present invention includes three types of information: trajectory points, the first frame of the video, that is, the input initial image and the entity mask.

[0122] The present invention proposes a position relationship perception module such as As shown in Figure 1, this module aims to focus on the spatial position and semantic information of objects in the input image and enrich the feature representation of objects. In the position-aware relationship module, for each object (entity) ,by Establish a polar coordinate system for the center and place adjacent objects The four-dimensional vector of the bounding box is the x-coordinate of the upper left corner of the bounding box; the y-coordinate of the upper left corner of the bounding box; the width w of the bounding box; the height h of the bounding box; Convert to polar coordinates The present invention recognizes that absolute position is rarely used when describing the positional relationship between objects. Instead, relative position is widely used (for example, object A is to the right of B, etc.). Therefore, the method of the present invention selects polar coordinate system to describe relative position instead of absolute position.

[0123] The present invention embeds the low-dimensional spatial position vector into a higher-dimensional space through a set of Gaussian kernels with learnable mean and covariance to facilitate subsequent calculations. The spatial relationship dimension after embedding Object by The Gaussian kernel operation centered on is defined as follows:

[0124] )

[0125] )

[0126] in, and is the Gaussian distribution of relative distances with learnable mean and covariance, and is a Gaussian distribution of angles with learnable mean and covariance.

[0127] The present invention aggregates objects through a scaling function and This means that the strength of the spatial relationship between objects can be weighted by the spatial direction. The aggregated spatial weight of the image is expressed as , defined as follows:

[0128]

[0129] in, is the number of objects in the image.

[0130] Another branch image feature pass Perform linear transformation. Semantic relationship Calculated according to the following formula. The present invention adopts a cross attention mechanism, where is the dimension of the feature.

[0131]

[0132] The relative position relationship between objects in the image is fused with the semantic relationship and scaled to the range of (0, 1) and calculated as follows:

[0133]

[0134] A multi-head attention mechanism is used to focus on the information in the input sequence from different angles and capture richer image feature information.

[0135]

[0136] k is the number of attention heads, which is set to 6.

[0137] The present invention uses image features , obtain the corresponding entity representation from the index coordinates in the segmentation mask. For ease of processing, an average pooling operation is used to obtain the final feature representation , where k represents the number of entities and the channel size is .

[0138] In order to associate these entity representations with the corresponding trajectory points, the present invention initializes a zero matrix , and then embed the entity vector according to the trajectory sequence points. During the training process, the present invention uses the entity mask of the first frame to extract the center coordinates of the entity , As the starting point of each trajectory sequence point. Through these center coordinate indexes, the entity representation can be embedded into the corresponding zero matrix In order to obtain the final entity representation .

[0139] Finally, the present invention introduces an encoder for fusion entity representation. The present invention uses four convolutional blocks to process the corresponding input guidance signal, each block consists of two convolutional layers and a SiLU activation function. Each block reduces the resolution of the input feature by half, and the final output resolution is 1 / 8. After passing through the encoder, the present invention adds the feature vector of the entity representation to the noise of the current frame according to the ControlNet method:

[0140]

[0141] in Indicates Potential noise in the frame.

[0142] Then, the feature The video is input into the encoder of the denoising 3D-Unet to obtain four features with different resolutions as latent conditional signals, and finally the video is reconstructed through the decoder.

[0143] The decoder uses the latent conditional signal and the features of the encoder to enhance the generated quality and retain key information. The specific workflow of the decoder is:

[0144] 1. Input latent conditional signals: The decoder receives the latent conditional signals obtained from the encoder, which are feature representations after dimensionality reduction.

[0145] 2. Upsampling: The decoder progressively upsamples these features to restore to the same resolution as the input video.

[0146] 3. Skip connection: During the upsampling process, the decoder can utilize the features of the corresponding layer in the encoder (through skip connections) to introduce high-resolution information into the decoding process to improve detail recovery capabilities.

[0147] 4. Generate video frames: Finally, the decoder outputs the reconstructed video frame, which is converted into pixel values ​​of the video frame through a final convolutional layer.

[0148] In summary, the decoder reconstructs the video by upsampling, using skip connections and non-linear activations, combining latent conditional signals and encoder features to ensure that the generated results have good quality and temporal consistency.

[0149] The following is a system embodiment corresponding to the above method embodiment. This embodiment can be implemented in conjunction with the above embodiment. The relevant technical details mentioned in the above embodiment are still valid in this embodiment. In order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the above embodiment.

[0150] like Figure 4 As shown, the present invention also proposes a trajectory guidance video generation device based on entity and position relationship, which includes:

[0151] The initial module obtains the video frame to be edited and its corresponding motion trajectory, wherein the video frame includes multiple entities; constructs an SVD model including a denoising diffusion model, an encoder and a decoder;

[0152] The extraction module establishes a polar coordinate system with each entity as the center, obtains the relative positions of the entities by converting the four-dimensional vectors of the bounding boxes of adjacent entities into polar coordinate vectors, and obtains the image features of the video frame by combining the relative positions with the entity semantic information of the video frame;

[0153] The representation module initializes a zero matrix corresponding to the size of the video frame, embeds the image features according to the trajectory sequence points of the motion trajectory, and obtains the entity representation;

[0154] The reconstruction module inputs the entity representation and the motion trajectory into the encoder to obtain the encoding result, superimposes the encoding result with Gaussian noise and inputs it into the denoising diffusion model to obtain multiple features of different resolutions as potential conditional signals, and finally decodes the potential conditional signal through the decoder to obtain the reconstructed video as the video generation result of the video frame under the motion trajectory.

[0155] The trajectory-guided video generation device based on entity and position relationship, wherein the trajectory sequence points of the motion trajectory are , ,in Indicates the length of the video.

[0156] The trajectory guidance video generation device based on entity and position relationship, wherein the extraction module comprises:

[0157] For each entity ,by Establish a polar coordinate system for the center and place the adjacent objects of i The 4D vector of the bounding box Convert to polar coordinates ;

[0158] The low-dimensional spatial position vector is embedded into a higher-dimensional space through a set of Gaussian kernels with learnable mean and covariance, and the entity i and The spatial relationship dimension after embedding ;entity by The Gaussian kernel operation centered on is defined as follows:

[0159] )

[0160] )

[0161] in, and is the Gaussian distribution of relative distances with learnable mean and covariance, and is the Gaussian distribution of angles with learnable mean and covariance;

[0162] Aggregate entities by a scaling function and The relative distance and angle relationship of the image, the aggregated spatial weight of the image is expressed as :

[0163]

[0164] in, is the number of entities in the video frame;

[0165] Image features pass Perform a linear transformation, is the dimension of the image feature characteristics;

[0166]

[0167] The relative position relationship between entities in the image is fused with the semantic relationship and scaled to the range of (0, 1) by the following formula:

[0168]

[0169] The multi-head attention mechanism is used to focus on the information in the input sequence from different angles, capturing richer image feature information;

[0170]

[0171] Where k is the number of attention heads.

[0172] By image features , obtain the corresponding entity representation from the index coordinates in the segmentation mask of the video frame; get the final image feature , where k represents the number of entities and the channel size is ;

[0173] The representation module includes: initializing a zero matrix , then embed the entity vector according to the trajectory sequence points, and use the entity mask of the video frame to extract the center coordinates of the entity , As the starting point of each trajectory sequence point, the entity representation is embedded into the corresponding zero matrix through the starting point coordinate index In order to obtain the final entity representation .

[0174] like Figure 5 As shown, the present invention further proposes a first electronic device A in another embodiment, including the above-mentioned trajectory guidance video generation device based on entity and position relationship.

[0175] like Figure 6 As shown, the first electronic device A can also be connected to the data acquisition device C and the information display device D through a wired or wireless information transmission scheme, the data acquisition device C is used to acquire video frames, such as the underwater video frames described in the embodiment of the present invention, and the information display device D is used to display the video generation results obtained by the analysis of the present invention.

[0176] The information display device D can sort and process the data output by the first electronic device A based on the information display mechanism to improve the readability of the data output by the first electronic device A. The information display mechanism can be manually preset, for example, the data output by the first electronic device A is visually displayed, which can be based on the display parameters and / or attributes set by the user. The display parameters can be, for example, the display data range, and the display attributes can be, for example, the display font, color, whether to scroll and play, etc. The user is presented with the key information specified by the user, and the user can understand the information more promptly without having to access the secondary page or scroll the page, saving the user's operation. Or the information display mechanism can be an artificial intelligence AI display model, which can learn the user's key information based on the user's previous usage habits, such as viewing time, number of clicks, number of edits, etc., and then automatically present rich and necessary key information to the user.

[0177] The present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a readable storage medium. When the computer program is executed by a processor, the computer can execute the trajectory-guided video generation method based on entity and position relationship provided by the above methods.

[0178] The present invention also proposes a storage medium VIII in another embodiment for storing a computer program for executing the trajectory guidance video generation method based on entity and position relationship. It should be understood that the storage medium in the embodiment of the present invention can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus RAM (DR RAM).

[0179] Figure 7A schematic block diagram of a second electronic device 1000 that can be used to implement an embodiment of the present invention is shown. The second electronic device 1000 electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The second electronic device 1000 can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples, and are not intended to limit the implementation of the present invention described and / or required herein. The second electronic device 1000 may be the same or different from the first electronic device A.

[0180] The second electronic device 1000 includes a computing unit I, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory II (ROM) or a computer program loaded from a storage medium VIII into a random access memory (RAM) III. In RAM III, various programs and data required for the operation of the device 1000 can also be stored. The computing unit I, ROM II, and RAM III are connected to each other via a bus IV. An input / output (I / O) interface V is also connected to the bus IV.

[0181] A plurality of components in the second electronic device 1000 are connected to the I / O interface V, including: an input unit VI, such as a keyboard, a mouse, etc.; an output unit VII, such as various types of displays, speakers, etc.; a storage medium VIII, such as a disk, an optical disk, etc.; and a communication unit IX, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit IX allows the second electronic device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0182] The computing unit I may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of computing unit I include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit I performs the various methods and processes described above, such as method steps S1-S4. For example, in some embodiments, the method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage medium VIII. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1000 via ROM II and / or communication unit IX. When the computer program is loaded into RAM III and executed by the computing unit I, one or more steps of the method described above may be performed. Alternatively, in other embodiments, the computing unit I may be configured to execute the method in any other appropriate manner (e.g., by means of firmware).

[0183] Although the embodiments of the present invention have been disclosed as above, they are not limited to the applications listed in the specification and the implementation modes, and they can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and the illustrations shown and described herein.

Claims

1. A trajectory-guided video generation method based on entity and position relationship, characterized in that: include: The initial step is to obtain a video frame to be edited and its corresponding motion trajectory, wherein the video frame includes multiple entities; and construct an SVD model including a denoising diffusion model, an encoder and a decoder; In the extraction step, a polar coordinate system is established with each entity as the center, and the relative positions between entities are obtained by converting the four-dimensional vectors of the bounding boxes of adjacent entities into polar coordinate vectors. The image features of the video frame are obtained by combining the relative positions with the entity semantic information of the video frame; In a representation step, a zero matrix corresponding to the size of the video frame is initialized, and the image features are embedded according to the trajectory sequence points of the motion trajectory to obtain an entity representation; In the reconstruction step, the entity representation and the motion trajectory are input into the encoder to obtain an encoding result, the encoding result is superimposed with Gaussian noise and then input into the denoising diffusion model to obtain multiple features of different resolutions as potential conditional signals, and finally the potential conditional signal is decoded by the decoder to obtain a reconstructed video as the video generation result of the video frame under the motion trajectory.

2. The method for generating a trajectory-guided video based on entity and position relationship according to claim 1, characterized in that: The trajectory sequence points of this motion trajectory are (x1, y1), (x2, y2),…, (x L ,y L ), where L represents the length of the video.

3. The method for generating a trajectory-guided video based on entity and position relationship according to claim 1, characterized in that: The extraction steps include: For each entity i, a polar coordinate system is established with i as the center, and the four-dimensional vector p of the bounding box of the neighboring object j of i is j =(x,y,w,h) converted to polar coordinate vector (p j ,θ j ); The low-dimensional spatial position vector is embedded into a higher-dimensional space through a set of Gaussian kernels with learnable mean and covariance. The embedded spatial relationship dimension d between entities i and j p =64; the Gaussian kernel operation of entity j centered on i is defined as follows: Among them, p0 and σ ρ is a Gaussian distribution of relative distances with learnable mean and covariance, θ0 and σ θ is the Gaussian distribution of angles with learnable mean and covariance; The relative distance and angle relationship between entities i and j are aggregated through a scaling function, and the aggregated spatial weight of the image is expressed as ω ρ : Where N is the number of entities in the video frame; The image feature V is linearly transformed by f(·), where d υ is the dimension of the image feature characteristics; The relative position relationship between entities in the image is fused with the semantic relationship and scaled to the range of (0,1) by the following formula: The multi-head attention mechanism is used to focus on the information in the input sequence from different angles, capturing richer image feature information; V r =f(V)+Concat[(ω I f(V))1,(ω I f(V))2,…,(ω I f(V)) k ] Where k is the number of attention heads. Through the image feature V r , obtain the corresponding entity representation from the index coordinates in the segmentation mask of the video frame; get the final image features {e1,e2,…,e k }, where k represents the number of entities and the channel size is C; The representation step includes: initializing a zero matrix Then, the entity vector is embedded according to the trajectory sequence points, and the entity mask of the video frame is used to extract the center coordinates of the entity x1,y1),(x2,y2),…,(x k ,y k ) is used as the starting point of each trajectory sequence point, and the entity representation is embedded into the corresponding zero matrix E through the starting point coordinate index, so as to obtain the final entity representation E'.

4. A trajectory guidance video generation device based on entity and position relationship, characterized in that: include: The initial module obtains the video frame to be edited and its corresponding motion trajectory, wherein the video frame includes multiple entities; constructs an SVD model including a denoising diffusion model, an encoder and a decoder; The extraction module establishes a polar coordinate system with each entity as the center, obtains the relative positions of the entities by converting the four-dimensional vectors of the bounding boxes of adjacent entities into polar coordinate vectors, and obtains the image features of the video frame by combining the relative positions with the entity semantic information of the video frame; The representation module initializes a zero matrix corresponding to the size of the video frame, embeds the image features according to the trajectory sequence points of the motion trajectory, and obtains the entity representation; The reconstruction module inputs the entity representation and the motion trajectory into the encoder to obtain the encoding result, superimposes the encoding result with Gaussian noise and inputs it into the denoising diffusion model to obtain multiple features of different resolutions as potential conditional signals, and finally decodes the potential conditional signal through the decoder to obtain the reconstructed video as the video generation result of the video frame under the motion trajectory.

5. The trajectory guidance video generation device based on entity and position relationship according to claim 4, characterized in that: The trajectory sequence points of this motion trajectory are (x1, y1), (x2, y2),…, (x L ,y L ), where L represents the length of the video.

6. The trajectory guidance video generation device based on entity and position relationship according to claim 4, characterized in that: The extraction module includes: For each entity i, a polar coordinate system is established with i as the center, and the four-dimensional vector p of the bounding box of the neighboring object j of i is j =(x,y,w,h) converted to polar coordinate vector (p j ,θ j ); The low-dimensional spatial position vector is embedded into a higher-dimensional space through a set of Gaussian kernels with learnable mean and covariance. The embedded spatial relationship dimension d between entities i and j p =64; the Gaussian kernel operation of entity j centered on i is defined as follows: Among them, p0 and σ ρ is a Gaussian distribution of relative distances with learnable mean and covariance, θ0 and σ θ is the Gaussian distribution of angles with learnable mean and covariance; The relative distance and angle relationship between entities i and j are aggregated through a scaling function, and the aggregated spatial weight of the image is expressed as ω ρ : Where N is the number of entities in the video frame; The image feature V is linearly transformed by f(·), where d υ is the dimension of the image feature characteristics; The relative position relationship between entities in the image is fused with the semantic relationship and scaled to the range of (0,1) by the following formula: The multi-head attention mechanism is used to focus on the information in the input sequence from different angles, capturing richer image feature information; V r =f(V)+Concat[(ω I f(V))1,(ω I f(V))2,…,(ω I f(V)) k ] Where k is the number of attention heads. Through the image feature V r , obtain the corresponding entity representation from the index coordinates in the segmentation mask of the video frame; get the final image features {e1,e2,…,e k }, where k represents the number of entities and the channel size is C; The representation module includes: initializing a zero matrix Then, the entity vector is embedded according to the trajectory sequence points, and the entity mask of the video frame is used to extract the center coordinates of the entity x1,y1),(x2,y2),…,(x k ,y k ) is used as the starting point of each trajectory sequence point, and the entity representation is embedded into the corresponding zero matrix E through the starting point coordinate index, so as to obtain the final entity representation E'.

7. An electronic device, characterized in that: It includes a trajectory guidance video generation device based on entity and position relationship as described in claim 4.

8. The electronic device according to claim 7, characterized in that: The electronic device is connected to an information display device, which is used to display the video generation result according to display parameters and attributes set by the user or through an artificial intelligence model.

9. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the method for generating a trajectory-guided video based on entity and position relationship described in any one of claims 1 to 3 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method for generating a trajectory-guided video based on entity and position relationship described in any one of claims 1 to 4 are implemented.

Images