An image-to-video generation method, device, computer equipment and storage medium

Abstract

The application relates to the field of image processing technology in financial and medical scenarios, and discloses an image-to-video generation method and device, computer equipment and a storage medium, the method comprising: receiving an input image and obtaining a corresponding multi-scale depth map; extracting a time-space feature sequence of the input image and the corresponding multi-scale depth map and performing coding and dynamic modeling to obtain six-degree-of-freedom motion transformation parameters; performing transformation and synthesis of continuous frames of the input image by using the six-degree-of-freedom motion transformation parameters to generate an initial output video sequence; performing authenticity discrimination on the initial output video sequence at a global time sequence level and a local time-space detail level respectively to generate an authenticity score; and iteratively performing optimization processing on the input image until the authenticity score reaches a preset threshold to generate a final output video sequence with high time-space coherence. The application has the advantages of significantly improving the time-space consistency and authenticity of the generated video.

Description

Technical Field

[0001] This invention relates to the field of image processing technology in financial and medical settings, and more particularly to an image-to-video generation method, apparatus, computer device, and storage medium. Background Technology

[0002] Image-to-video generation technology is an important research direction at the intersection of computer vision and artificial intelligence, and has shown broad application potential in various industries in recent years. Traditional methods mainly rely on models such as generative adversarial networks (GANs) and variational autoencoders (VAEs) to achieve video synthesis by separating content and motion components. For example, MoCoGAN generates video sequences by modeling motion and content latent spaces separately, but its modeling of temporal consistency still heavily relies on a large amount of labeled data, which limits its promotion and application in real-world scenarios.

[0003] In practical applications, such as in the financial sector, dynamic video generation technology can be used for virtual customer service, product visualization demonstrations, and remote identity authentication. However, it demands extremely high levels of video smoothness, realism, and stability. Existing technologies often fail to meet the financial industry's requirements for a reliable interactive experience due to issues like image jitter and motion distortion. In the medical field, generating dynamic simulation videos from single medical images (such as disease progression simulations, surgical planning, and medical teaching) is of significant value. However, medical data annotation is costly and subject to strong privacy constraints, limiting the applicability of traditional supervised learning methods. Furthermore, the generated videos must strictly conform to anatomical structures and motion patterns, and existing methods still have significant shortcomings in modeling complex biological movements.

[0004] Therefore, there is an urgent need for a generation framework that does not rely on labeled data, can collaboratively optimize depth information and motion trajectory, and has strong spatiotemporal consistency constraints, in order to support the spatiotemporal coherence and realism of video generation technology in high-reliability fields such as finance and healthcare, and ensure the quality and credibility of the generated videos. Summary of the Invention

[0005] The purpose of this invention is to provide an image-to-video generation method, apparatus, device, and storage medium, aiming to solve the problem of how to improve the spatiotemporal coherence and realism of video during the image-to-video generation process.

[0006] In a first aspect, embodiments of the present invention provide an image-to-video generation method, comprising:

[0007] The input image is received by a depth perception generator and its multi-scale visual features are extracted to obtain a multi-scale depth map corresponding to the input image.

[0008] The spatiotemporal feature sequences of the input image and its corresponding multi-scale depth map are extracted by a motion trajectory predictor, and the spatiotemporal feature sequences are encoded and dynamically modeled to obtain six-degree-of-freedom motion transformation parameters for representing the rigid motion of the scene.

[0009] The input image is transformed and synthesized frame by frame using the six-degree-of-freedom motion transformation parameters by the differentiable image warping module to generate an initial output video sequence.

[0010] The spatiotemporal discriminator uses the initial output video sequence to perform authenticity discrimination at both the global temporal level and the local spatiotemporal detail level, respectively, and generates an authenticity score.

[0011] The depth perception generator, motion trajectory predictor, and spatiotemporal discriminator are simultaneously optimized using this joint loss function; wherein the joint loss function includes at least a constraint on the depth of adjacent frames. Figure 1 Consistent contrast depth loss and adversarial loss used to constrain the realism of generated videos;

[0012] The input image is processed iteratively until the authenticity score reaches a preset threshold, generating a final output video sequence with high spatiotemporal coherence.

[0013] Secondly, embodiments of the present invention provide an image-to-video generation apparatus, comprising:

[0014] The feature extraction unit is used to receive the input image through the depth perception generator and extract its multi-scale visual features to obtain a multi-scale depth map corresponding to the input image.

[0015] The encoding and modeling unit is used to extract the spatiotemporal feature sequence of the input image and its corresponding multi-scale depth map through the motion trajectory predictor, and to encode and dynamically model the spatiotemporal feature sequence to obtain six-degree-of-freedom motion transformation parameters for representing the rigid motion of the scene.

[0016] The initial generation unit is used to transform and synthesize the input image frame by frame using the six-degree-of-freedom motion transformation parameters through the differentiable image warping module to generate an initial output video sequence;

[0017] The scoring unit is used to perform authenticity discrimination at the global temporal level and the local spatiotemporal detail level using the initial output video sequence through a spatiotemporal discriminator, and generate an authenticity score.

[0018] An optimization unit is configured to simultaneously optimize the depth-sensing generator, motion trajectory predictor, and spatiotemporal discriminator using the joint loss function; wherein the joint loss function includes at least elements for constraining the depth of adjacent frames. Figure 1Consistent contrast depth loss and adversarial loss used to constrain the realism of generated videos;

[0019] The final output unit is used to iteratively process the input image until the authenticity score reaches a preset threshold, generating a final output video sequence with high spatiotemporal coherence.

[0020] Thirdly, embodiments of the present invention provide a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the image-to-video generation method described in the first aspect above.

[0021] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the image-to-video generation method described in the first aspect.

[0022] In the aforementioned image-to-video generation method, apparatus, device, and storage medium, a depth-sensing generator receives an input image and extracts its multi-scale visual features to obtain a multi-scale depth map corresponding to the input image. A motion trajectory predictor extracts the spatiotemporal feature sequence of the input image and its corresponding multi-scale depth map, and encodes and dynamically models the spatiotemporal feature sequence to obtain six-degree-of-freedom motion transformation parameters representing the rigid motion of the scene. A differentiable image warping module uses the six-degree-of-freedom motion transformation parameters to transform and synthesize consecutive frames of the input image, generating an initial output video sequence. A spatiotemporal discriminator uses the initial output video sequence to perform authenticity discrimination at both the global temporal level and the local spatiotemporal detail level, generating an authenticity score. The joint loss function simultaneously optimizes the depth-sensing generator, motion trajectory predictor, and spatiotemporal discriminator. The joint loss function includes at least elements for constraining the depth of adjacent frames. Figure 1The method employs a contrastive depth loss and an adversarial loss to constrain the realism of the generated video. It iteratively processes the input image until the realism score reaches a preset threshold, generating a final output video sequence with high spatiotemporal coherence. This invention provides an overall workflow for image-to-video generation in financial and medical scenarios, including key steps such as depth-aware generation, motion trajectory prediction, differentiable image distortion, spatiotemporal discriminator realism assessment, and joint loss optimization. This method significantly improves the spatiotemporal consistency and realism of the generated video through a self-supervised learning mechanism, utilizing the joint optimization of contrastive depth loss and adversarial loss. In financial scenarios, this method can be used to generate dynamic product display videos or virtual customer service interaction videos, ensuring smooth, jitter-free visuals and enhancing customer experience and trust. In medical scenarios, it can generate disease deduction or surgical simulation videos from single medical images (such as CT or MRI slices), assisting doctors in visual diagnosis and teaching, while avoiding high dependence on labeled data and meeting medical data privacy and security requirements. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram of an application environment for the image-to-video generation method provided in an embodiment of the present invention;

[0025] Figure 2 This is a flowchart illustrating an image-to-video generation method provided in an embodiment of the present invention;

[0026] Figure 3 This is a schematic block diagram of an image-to-video generation apparatus provided in an embodiment of the present invention;

[0027] Figure 4 A schematic diagram of a computer device is provided for an embodiment of the present invention;

[0028] Figure 5 Another structural schematic diagram of a computer device is provided for an embodiment of the present invention. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] The image-to-video generation method provided in this invention can be applied to, for example... Figure 1 In this application environment, the client communicates with the server via a network. The client can receive the input image through a depth-sensing generator and extract its multi-scale visual features to obtain a multi-scale depth map of the corresponding input image. A motion trajectory predictor extracts the spatiotemporal feature sequence of the input image and its corresponding multi-scale depth map, and encodes and dynamically models the spatiotemporal feature sequence to obtain six-degree-of-freedom motion transformation parameters representing the rigid motion of the scene. A differentiable image warping module uses these six-degree-of-freedom motion transformation parameters to transform and synthesize consecutive frames of the input image, generating an initial output video sequence. A spatiotemporal discriminator uses the initial output video sequence to perform realism judgment at both the global temporal level and the local spatiotemporal detail level, generating a realism score. The joint loss function simultaneously optimizes the depth-sensing generator, motion trajectory predictor, and spatiotemporal discriminator; the joint loss function includes at least elements for constraining the depth of adjacent frames. Figure 1 This invention provides an overall process for image-to-video generation, including key steps such as depth-aware generation, motion trajectory prediction, differentiable image distortion, spatiotemporal discriminator realism assessment, and joint loss optimization. This method significantly improves the spatiotemporal consistency and realism of the generated video through a self-supervised learning mechanism, utilizing contrastive depth loss and adversarial loss for joint optimization. In financial scenarios, this method can be used to generate dynamic product display videos or virtual customer service interaction videos, ensuring smooth, jitter-free visuals and enhancing customer experience and trust. In medical scenarios, it can generate disease deduction or surgical simulation videos from single medical images (such as CT or MRI slices), assisting doctors in visual diagnosis and teaching, while avoiding high dependence on labeled data and meeting medical data privacy and security requirements. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The present invention will now be described in detail through specific embodiments.

[0031] Please see Figure 2 As shown, Figure 2 This is a flowchart illustrating the image-to-video generation method provided in an embodiment of the present invention.

[0032] like Figure 2 As shown, the method includes steps S201 to S206.

[0033] S201. The input image is received by the depth perception generator and its multi-scale visual features are extracted to obtain the multi-scale depth map of the corresponding input image.

[0034] Step S201 aims to extract multi-scale depth information from the input image using a depth-aware generator, providing geometric priors for subsequent motion modeling.

[0035] In one embodiment, step S201 includes:

[0036] The input image is segmented into a sequence of image patches and multi-scale hierarchical features are extracted using a visual Transformer-based encoder.

[0037] The multi-scale hierarchical features are upsampled and fused by a decoder containing skip connections to obtain fused features.

[0038] By processing fused features through convolutional layers and activation functions, a multi-scale depth map is output.

[0039] This embodiment defines the specific implementation of the depth-aware generator, employing an encoder-decoder structure based on the Visual Transformer (ViT). The encoder divides the input image into a 16×16 sequence and extracts multi-level features through a multi-head self-attention mechanism; the decoder fuses multi-scale features through skip connections, and finally outputs a normalized multi-scale depth map through 1×1 convolution and a sigmoid activation function.

[0040] More specifically, the representation of multi-scale depth maps is as follows:

[0041]

[0042] in, Let i represent the multi-scale depth map of the current frame t, where i and j are the row and column of the depth feature map, respectively. Indicates the depth prediction value; z max =100m,z min =0.1m, which is within the preset depth range.

[0043] This embodiment enables the model to capture depth information from minute local details to the global scene. Especially in medical imaging, it can accurately distinguish the depth layers of different tissue structures (such as tumors and surrounding tissues), providing precise geometric constraints for motion modeling. In financial scenarios, this technology can enhance the three-dimensionality and spatial rationality of virtual product displays, improving the realism and trustworthiness of the user experience.

[0044] S202. Extract the spatiotemporal feature sequences of the input image and its corresponding multi-scale depth map through the motion trajectory predictor, and encode and dynamically model the spatiotemporal feature sequences to obtain six-degree-of-freedom motion transformation parameters for representing the rigid motion of the scene.

[0045] Step S202 aims to use a motion trajectory predictor to perform spatiotemporal dynamic modeling based on spatiotemporal feature sequences (i.e. RGB-D feature sequences) and 3D convolutional LSTM to output six degrees of freedom motion parameters that conform to physical laws.

[0046] In one embodiment, step S202 includes:

[0047] The input image and the features of the corresponding multi-scale depth map are concatenated to form a spatiotemporal feature sequence, where the spatiotemporal feature sequence is RGB-D features;

[0048] A 3D convolutional long short-term memory network is used to perform spatiotemporal modeling of spatiotemporal feature sequences, and cell states are updated through a gating mechanism to capture dynamic dependencies.

[0049] The six-degree-of-freedom motion parameters are regressed from the final hidden state through a fully connected layer. These six-degree-of-freedom motion parameters include a three-dimensional rotation vector and a three-dimensional translation vector.

[0050] This embodiment clarifies the core structure and working mechanism of the motion trajectory predictor. This module concatenates the input image with a multi-scale depth map to form a spatiotemporal feature sequence (RGB-D feature sequence), which is then input into a 5-layer ConvLSTM3D network for spatiotemporal modeling. Cell states are dynamically updated through gating mechanisms such as forget gates, input gates, and output gates to capture motion dependencies in the scene. Finally, a multilayer perceptron (MLP) regresses six-degree-of-freedom motion parameters (3D rotation vector and 3D translation vector) from the hidden states. This ensures that motion prediction conforms to the rigid motion laws of the physical world, avoiding the generation of unreasonable or distorted dynamic effects. Based on this, in medical scenarios, this embodiment can be used to simulate the motion trajectory of surgical instruments or the dynamic changes of organs, ensuring compliance with anatomical constraints; in financial scenarios, it can be used to generate dynamic data charts or interactive product demonstrations, enhancing the accuracy and appeal of information delivery.

[0051] S203. The input image is transformed and synthesized frame by frame using the six-degree-of-freedom motion transformation parameters through the differentiable image warping module to generate the initial output video sequence.

[0052] Step S203 aims to synthesize consecutive frames through a differentiable image warping module, ensuring that the transformation process is differentiable and optimizable.

[0053] In one embodiment, step S203 includes:

[0054] Calculate the corresponding rigid transformation matrix based on the six degrees of freedom motion parameters at each time step;

[0055] The rigid transformation matrix is ​​used to perform differentiable sampling and coordinate mapping on the input image or the image generated from the previous frame in order to synthesize the image content of the current frame;

[0056] The image content is subjected to continuous and physically consistent rigid transformation and image synthesis to generate an initial output video frame with temporal coherence;

[0057] The initial output video frames generated at each time step are combined sequentially to obtain the initial output video sequence.

[0058] This embodiment defines the implementation mechanism of the differentiable image distortion module. This module calculates a rigid transformation matrix based on the six-degree-of-freedom motion parameters at each time step, and performs coordinate mapping and resampling on the input image or the previous frame image through a differentiable bilinear sampling operation to synthesize the image content of the current frame. Then, by performing precise rigid transformation and synthesis on the image content of each frame, it ensures the temporal coherence and physical consistency of each initial output video frame. This process not only maintains the physical rationality of the geometric transformation but also ensures gradient backpropagation, enabling end-to-end training of the entire system. Based on this, in medical video generation, this embodiment can ensure smooth and natural movement of organs or lesions, avoiding visual breaks or distortions; in financial scenarios, it can ensure consistent and coherent movement of elements in dynamic visualization content, improving professionalism and credibility.

[0059] S204. Using a spatiotemporal discriminator, the initial output video sequence is used to determine authenticity at both the global temporal level and the local spatiotemporal detail level, generating an authenticity score.

[0060] Step S204 aims to perform authenticity determination at both the global and local levels using a dual-path spatiotemporal discriminator.

[0061] In one embodiment, step S204 includes:

[0062] The overall temporal coherence and authenticity of the initial output video sequence are processed and analyzed by a 3D convolutional neural network in the global discriminant path, and a global authenticity score is output.

[0063] The local discriminant path uses a patch-based convolutional neural network to determine the authenticity of action and texture details in spatiotemporal local regions of the initial output video sequence and outputs a local authenticity score.

[0064] By combining the global authenticity score and the local authenticity score, an authenticity score is obtained for the initial output video sequence, which serves as the discriminator.

[0065] This embodiment defines a dual-path structure and discrimination mechanism for the spatiotemporal discriminator. The global discrimination path uses 3DResNet-18 to process 16×224×224 video blocks (i.e., the segmented initial output video sequence) and outputs a global realism score. The local discrimination path uses a PatchGAN structure to discriminate 32×32 spatiotemporal regions (i.e., the segmented spatiotemporal local regions in the initial output video sequence) and outputs a local realism score. This multi-scale discrimination strategy can simultaneously ensure the coherence of the generated video over long time and the realism of local motion details. Based on this, in medical applications, this embodiment can detect abnormal movements that do not conform to physiological laws, such as organ shaking or motion distortion; in financial scenarios, it can ensure the naturalness and smoothness of dynamic interactive elements such as floating buttons and data animations, improving the user interaction experience.

[0066] S205. Simultaneously optimize the depth perception generator, motion trajectory predictor, and spatiotemporal discriminator using the joint loss function; wherein the joint loss function includes at least one function for constraining the depth of adjacent frames. Figure 1 Consistent contrast depth loss and adversarial loss used to constrain the realism of generated videos.

[0067] Step S205 aims to achieve overall system optimization through a joint loss function (contrast depth loss + adversarial loss).

[0068] In one embodiment, step S205 includes:

[0069] A joint loss function consisting of contrastive depth loss and adversarial loss is constructed. The contrastive depth loss constrains the model to learn geometric consistency by maximizing the similarity of depth features between adjacent frames in the same video sequence and minimizing the similarity of features with non-adjacent frames. The adversarial loss uses the authenticity score provided by the spatiotemporal discriminator to distinguish between the real video sequence corresponding to the input image and the generated initial output video sequence and calculates the loss accordingly.

[0070] By combining the loss function and updating the parameters of the depth perception generator, motion trajectory predictor, and spatiotemporal discriminator in an end-to-end manner, the model converges.

[0071] This embodiment defines the construction and optimization mechanism of the joint loss function. Contrastive depth loss, through the InfoNCE loss function, maximizes the similarity of depth features between adjacent frames and minimizes the similarity with non-adjacent frames, forcing the model to learn a temporally consistent depth representation. Adversarial loss, through a variant of Hinge Loss, distinguishes between real video sequences and the generated initial output video sequence, improving overall visual realism. The combination of these two approaches enables the system to learn a highly consistent spatiotemporal representation even under unsupervised conditions. In medical scenarios, this loss function design reduces reliance on expensive labeled data while ensuring the generated content conforms to medical rationale; in financial scenarios, it improves the professional quality of the generated videos and the consistency of user experience.

[0072] More specifically, compare depth loss Represented as:

[0073]

[0074] Where N represents the number of positive sample pairs; K represents the number of negative sample pairs; Indicates the similarity between positive sample pairs. E(D t ) and E(D t+1 () represent the depth maps of the preceding and following frames, respectively; For negative sample pairs (D) t Non-D t+1 Similarity, where τ represents the temperature coefficient, and τ can be 0.07.

[0075] More specifically, combating losses Represented as:

[0076]

[0077] Where E represents the expected value. This indicates the discriminator's score for the corresponding real video sequence. This represents the discriminator's score for the initial output video sequence.

[0078] S206. Iteratively process the input image (i.e., the process of steps S201-S205 above) until the authenticity score reaches the preset threshold, and generate the final output video sequence with high spatiotemporal coherence.

[0079] Step S206 aims to continuously optimize the generated video sequence by repeatedly executing the aforementioned steps of depth perception generation, motion trajectory prediction, and spatiotemporal discrimination; ensuring that the output of each frame is as close as possible to the temporal and depth features of the real video, thereby making the generated video sequence more visually coherent and realistic.

[0080] The preset threshold in step S206 can be set based on a large amount of experimental data and user feedback, aiming to balance computational efficiency and output quality, and ensure that the final video generation result not only meets the needs of practical applications, but also maintains a high degree of consistency under unsupervised conditions.

[0081] In one embodiment, step S206 includes:

[0082] In each iteration, the currently generated output video sequence and its corresponding depth map sequence are input into the spatiotemporal discriminator again to obtain a new round of authenticity score;

[0083] The new round of authenticity score is compared with the preset threshold. If the threshold is not reached, the gradient calculated by the joint loss function is backpropagated and the parameters of at least one module in the depth perception generator, motion trajectory predictor and spatiotemporal discriminator are updated.

[0084] The updated model is used to re-execute the entire process of input image processing until the authenticity score output by the spatiotemporal discriminator reaches a preset threshold, and the video sequence generated in the current iteration is used as the final output video sequence.

[0085] This embodiment clarifies the specific process of the iterative optimization mechanism. Specifically, in each iteration, the system re-inputs the currently generated output video sequence into the spatiotemporal discriminator to obtain a realism score. If the score does not reach a preset threshold, the gradient is calculated using the joint loss function, and the model parameters are updated in reverse. This mechanism ensures that the model has good convergence and adaptability, and can dynamically adjust the generated effect according to the quality requirements of different application scenarios. In medical scenarios, this mechanism can adapt to the output characteristics of different imaging equipment (such as CT and MRI); in financial scenarios, the quality standards and style of the generated video can be adjusted according to business needs (such as virtual customer service and product demonstrations).

[0086] As can be seen, the above solution provides a complete workflow for image-to-video generation in financial and medical scenarios, including key steps such as depth-sensing generation, motion trajectory prediction, differentiable image warping, spatiotemporal discriminator realism assessment, and joint loss optimization. This method significantly improves the spatiotemporal consistency and realism of the generated video through a self-supervised learning mechanism, utilizing contrastive depth loss and adversarial loss for joint optimization. In financial scenarios, this method can be used to generate dynamic product demonstration videos or virtual customer service interaction videos, ensuring smooth, jitter-free visuals and enhancing customer experience and trust. In medical scenarios, it can generate disease deduction or surgical simulation videos from single medical images (such as CT or MRI slices), assisting doctors in visual diagnosis and teaching, while avoiding high dependence on labeled data and complying with medical data privacy and security requirements.

[0087] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0088] In one embodiment, an image-to-video generation apparatus is provided, which corresponds one-to-one with the image-to-video generation method described in the above embodiments. For example... Figure 3 As shown, the image-to-video generation device includes a feature extraction unit 301, an encoding and modeling unit 302, an initial generation unit 303, a scoring unit 304, an optimization unit 305, and a final output unit 306. Detailed descriptions of each functional module are as follows:

[0089] The feature extraction unit 301 is used to receive the input image through the depth perception generator and extract its multi-scale visual features to obtain the multi-scale depth map of the corresponding input image.

[0090] The encoding and modeling unit 302 is used to extract the spatiotemporal feature sequence of the input image and its corresponding multi-scale depth map through the motion trajectory predictor, and to encode and dynamically model the spatiotemporal feature sequence to obtain six-degree-of-freedom motion transformation parameters for representing the rigid motion of the scene.

[0091] The initial generation unit 303 is used to transform and synthesize the input image frame by frame using the six-degree-of-freedom motion transformation parameters through the differentiable image warping module to generate an initial output video sequence.

[0092] The scoring unit 304 is used to perform authenticity discrimination at the global temporal level and the local spatiotemporal detail level using the initial output video sequence through a spatiotemporal discriminator, and generate an authenticity score.

[0093] Optimization unit 305 is used to simultaneously optimize the depth perception generator, motion trajectory predictor, and spatiotemporal discriminator by utilizing the joint loss function; wherein the joint loss function includes at least the function for constraining the depth of adjacent frames. Figure 1 Consistent contrast depth loss and adversarial loss used to constrain the realism of generated videos;

[0094] The final output unit 306 is used to iteratively process the input image until the authenticity score reaches a preset threshold, generating a final output video sequence with high spatiotemporal coherence.

[0095] In one embodiment, the feature extraction unit 301 is specifically used for:

[0096] The input image is segmented into a sequence of image patches and multi-scale hierarchical features are extracted using a visual Transformer-based encoder.

[0097] The multi-scale hierarchical features are upsampled and fused by a decoder containing skip connections to obtain fused features.

[0098] By processing fused features through convolutional layers and activation functions, a multi-scale depth map is output.

[0099] In one embodiment, the encoding and modeling unit 302 is specifically used for:

[0100] The input image and the features of the corresponding multi-scale depth map are concatenated to form a spatiotemporal feature sequence, where the spatiotemporal feature sequence is RGB-D features;

[0101] A 3D convolutional long short-term memory network is used to perform spatiotemporal modeling of spatiotemporal feature sequences, and cell states are updated through a gating mechanism to capture dynamic dependencies.

[0102] The six-degree-of-freedom motion parameters are regressed from the final hidden state through a fully connected layer. These six-degree-of-freedom motion parameters include a three-dimensional rotation vector and a three-dimensional translation vector.

[0103] In one embodiment, the initial generation unit 303 is specifically used for:

[0104] Calculate the corresponding rigid transformation matrix based on the six degrees of freedom motion parameters at each time step;

[0105] The rigid transformation matrix is ​​used to perform differentiable sampling and coordinate mapping on the input image or the image generated from the previous frame in order to synthesize the image content of the current frame;

[0106] The image content is subjected to continuous and physically consistent rigid transformation and image synthesis to generate an initial output video frame with temporal coherence;

[0107] The initial output video frames generated at each time step are combined sequentially to obtain the initial output video sequence.

[0108] In one embodiment, the scoring unit 304 is specifically used for:

[0109] The overall temporal coherence and authenticity of the initial output video sequence are processed and analyzed by a 3D convolutional neural network in the global discriminant path, and a global authenticity score is output.

[0110] The local discriminant path uses a patch-based convolutional neural network to determine the authenticity of action and texture details in spatiotemporal local regions of the initial output video sequence and outputs a local authenticity score.

[0111] By combining the global authenticity score and the local authenticity score, an authenticity score is obtained for the initial output video sequence, which serves as the discriminator.

[0112] In one embodiment, the optimization unit 305 is specifically used for:

[0113] A joint loss function consisting of contrastive depth loss and adversarial loss is constructed. The contrastive depth loss constrains the model to learn geometric consistency by maximizing the similarity of depth features between adjacent frames in the same video sequence and minimizing the similarity of features with non-adjacent frames. The adversarial loss uses the authenticity score provided by the spatiotemporal discriminator to distinguish between the real video sequence corresponding to the input image and the generated initial output video sequence and calculates the loss accordingly.

[0114] By combining the loss function and updating the parameters of the depth perception generator, motion trajectory predictor, and spatiotemporal discriminator in an end-to-end manner, the model converges.

[0115] In one embodiment, the final output unit 306 is specifically used for:

[0116] In each iteration, the currently generated output video sequence and its corresponding depth map sequence are input into the spatiotemporal discriminator again to obtain a new round of authenticity score;

[0117] The new round of authenticity score is compared with the preset threshold. If the threshold is not reached, the gradient calculated by the joint loss function is backpropagated and the parameters of at least one module in the depth perception generator, motion trajectory predictor and spatiotemporal discriminator are updated.

[0118] The updated model is used to re-execute the entire process of input image processing until the authenticity score output by the spatiotemporal discriminator reaches a preset threshold, and the video sequence generated in the current iteration is used as the final output video sequence.

[0119] Specific limitations regarding the image-to-video generation apparatus can be found in the limitations of the image-to-video generation method described above, and will not be repeated here. Each module in the aforementioned image-to-video generation apparatus can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the processor in a computer device, or stored in software in the memory of a computer device, so that the processor can call and execute the operations corresponding to each module.

[0120] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When executed by the processor, the computer program implements the functions or steps of an image-to-video generation method on the server side.

[0121] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements the functions or steps of an image-to-video generation method on the client side.

[0122] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0123] The input image is received by a depth perception generator and its multi-scale visual features are extracted to obtain a multi-scale depth map of the corresponding input image.

[0124] The spatiotemporal feature sequences of the input image and its corresponding multi-scale depth map are extracted by a motion trajectory predictor, and the spatiotemporal feature sequences are encoded and dynamically modeled to obtain six-degree-of-freedom motion transformation parameters to represent the rigid motion of the scene.

[0125] The input image is transformed and synthesized frame by frame using a differentiable image warping module with six degrees of freedom motion transformation parameters to generate an initial output video sequence.

[0126] The spatiotemporal discriminator uses the initial output video sequence to determine authenticity at both the global temporal level and the local spatiotemporal detail level, generating an authenticity score.

[0127] This joint loss function simultaneously optimizes the depth-sensing generator, motion trajectory predictor, and spatiotemporal discriminator; wherein the joint loss function includes at least elements for constraining the depth of adjacent frames. Figure 1 Consistent contrast depth loss and adversarial loss used to constrain the realism of generated videos;

[0128] The input image is processed iteratively until the authenticity score reaches a preset threshold, generating a final output video sequence with high spatiotemporal coherence.

[0129] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0130] The input image is received by a depth perception generator and its multi-scale visual features are extracted to obtain a multi-scale depth map of the corresponding input image.

[0131] The spatiotemporal feature sequences of the input image and its corresponding multi-scale depth map are extracted by a motion trajectory predictor, and the spatiotemporal feature sequences are encoded and dynamically modeled to obtain six-degree-of-freedom motion transformation parameters to represent the rigid motion of the scene.

[0132] The input image is transformed and synthesized frame by frame using a differentiable image warping module with six degrees of freedom motion transformation parameters to generate an initial output video sequence.

[0133] The spatiotemporal discriminator uses the initial output video sequence to determine authenticity at both the global temporal level and the local spatiotemporal detail level, generating an authenticity score.

[0134] This joint loss function simultaneously optimizes the depth-sensing generator, motion trajectory predictor, and spatiotemporal discriminator; wherein the joint loss function includes at least elements for constraining the depth of adjacent frames. Figure 1 Consistent contrast depth loss and adversarial loss used to constrain the realism of generated videos;

[0135] The input image is processed iteratively until the authenticity score reaches a preset threshold, generating a final output video sequence with high spatiotemporal coherence.

[0136] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0137] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0138] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0139] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. An image-to-video generation method, characterized in that, include: The input image is received by a depth perception generator and its multi-scale visual features are extracted to obtain a multi-scale depth map corresponding to the input image. The spatiotemporal feature sequences of the input image and its corresponding multi-scale depth map are extracted by a motion trajectory predictor, and the spatiotemporal feature sequences are encoded and dynamically modeled to obtain six-degree-of-freedom motion transformation parameters for representing the rigid motion of the scene. The input image is transformed and synthesized frame by frame using the six-degree-of-freedom motion transformation parameters by the differentiable image warping module to generate an initial output video sequence. The spatiotemporal discriminator uses the initial output video sequence to perform authenticity discrimination at both the global temporal level and the local spatiotemporal detail level, respectively, and generates an authenticity score. The depth-aware generator, motion trajectory predictor, and spatiotemporal discriminator are simultaneously optimized by utilizing a joint loss function; wherein the joint loss function includes at least a contrastive depth loss for constraining the consistency of depth maps in adjacent frames and an adversarial loss for constraining the realism of the generated video; The input image is processed iteratively until the authenticity score reaches a preset threshold, generating a final output video sequence with high spatiotemporal coherence.

2. The image-to-video generation method according to claim 1, characterized in that: The step of receiving the input image through a depth-sensing generator and extracting its multi-scale visual features to obtain a multi-scale depth map corresponding to the input image includes: The input image is segmented into a sequence of image patches and multi-scale hierarchical features are extracted using a visual Transformer-based encoder. The multi-scale hierarchical features are upsampled and fused using a decoder that includes skip connections to obtain fused features. The fused features are processed by convolutional layers and activation functions to output a multi-scale depth map.

3. The image-to-video generation method according to claim 1, characterized in that: The process involves extracting the spatiotemporal feature sequences of the input image and its corresponding multi-scale depth map using a motion trajectory predictor, and encoding and dynamically modeling the spatiotemporal feature sequences to obtain six-degree-of-freedom motion transformation parameters representing the rigid motion of the scene, including: The input image is concatenated with the features of the corresponding multi-scale depth map to form a spatiotemporal feature sequence, wherein the spatiotemporal feature sequence is RGB-D feature; The spatiotemporal feature sequence is spatiotemporally modeled using a 3D convolutional long short-term memory network, and the cell state is updated through a gating mechanism to capture dynamic dependencies. The six degrees of freedom motion parameters are regressed from the final hidden state through a fully connected layer. The six degrees of freedom motion parameters include a three-dimensional rotation vector and a three-dimensional translation vector.

4. The image-to-video generation method according to claim 1, characterized in that: The step of using the differentiable image warping module to transform and synthesize consecutive frames of the input image using the six-degree-of-freedom motion transformation parameters to generate an initial output video sequence includes: Calculate the corresponding rigid transformation matrix based on the six degrees of freedom motion parameters at each time step; The rigid transformation matrix is ​​used to perform differentiable sampling and coordinate mapping on the input image or the image generated from the previous frame to synthesize the image content of the current frame; The image content is subjected to continuous and physically consistent rigid transformation and image synthesis to generate an initial output video frame with temporal coherence; The initial output video frames generated at each time step are combined sequentially to obtain the initial output video sequence.

5. The image-to-video generation method according to claim 1, characterized in that: The process of using a spatiotemporal discriminator to perform authenticity assessments on the initial output video sequence at both the global temporal level and the local spatiotemporal detail level, and generating an authenticity score, includes: The overall temporal coherence and authenticity of the initial output video sequence are processed and analyzed by a 3D convolutional neural network in the global discrimination path, and a global authenticity score is output. The local discriminant path uses a patch-based convolutional neural network to determine the authenticity of action and texture details in spatiotemporal local regions of the initial output video sequence, and outputs a local authenticity score. By combining the global authenticity score and the local authenticity score, an authenticity score for the initial output video sequence is obtained, which serves as the discriminator.

6. The image-to-video generation method according to claim 1, characterized in that: The method utilizes the joint loss function to simultaneously optimize the depth-aware generator, motion trajectory predictor, and spatiotemporal discriminator; wherein the joint loss function includes at least a contrastive depth loss for constraining the consistency of depth maps in adjacent frames and an adversarial loss for constraining the realism of the generated video, including: A joint loss function consisting of contrastive depth loss and adversarial loss is constructed. The contrastive depth loss constrains the model to learn geometric consistency by maximizing the similarity of depth features between adjacent frames in the same video sequence and minimizing the similarity of features with non-adjacent frames. The adversarial loss distinguishes the real video sequence corresponding to the input image from the generated initial output video sequence based on the authenticity score provided by the spatiotemporal discriminator and calculates the loss accordingly. The parameters of the depth perception generator, motion trajectory predictor, and spatiotemporal discriminator are iteratively updated using the joint loss function in an end-to-end manner until the model reaches the convergence condition.

7. The image-to-video generation method according to claim 1, characterized in that: The iterative processing of the input image continues until the authenticity score reaches a preset threshold, generating a final output video sequence with high spatiotemporal coherence, including: In each iteration, the currently generated output video sequence and its corresponding depth map sequence are input back into the spatiotemporal discriminator to obtain a new round of authenticity score; The new round of authenticity score is compared with the preset threshold. If the preset threshold is not reached, the gradient calculated by the joint loss function is backpropagated and the parameters of at least one module in the depth perception generator, motion trajectory predictor and spatiotemporal discriminator are updated. The updated model is used to re-execute the entire process of the input image until the authenticity score output by the spatiotemporal discriminator reaches the preset threshold, and the video sequence generated in the current iteration is used as the final output video sequence.

8. An image-to-video generation apparatus, characterized in that, include: The feature extraction unit is used to receive the input image through the depth perception generator and extract its multi-scale visual features to obtain a multi-scale depth map corresponding to the input image. The encoding and modeling unit is used to extract the spatiotemporal feature sequence of the input image and its corresponding multi-scale depth map through the motion trajectory predictor, and to encode and dynamically model the spatiotemporal feature sequence to obtain six-degree-of-freedom motion transformation parameters for representing the rigid motion of the scene. The initial generation unit is used to transform and synthesize the input image frame by frame using the six-degree-of-freedom motion transformation parameters through the differentiable image warping module to generate an initial output video sequence; The scoring unit is used to perform authenticity discrimination at the global temporal level and the local spatiotemporal detail level using the initial output video sequence through a spatiotemporal discriminator, and generate an authenticity score. An optimization unit is configured to simultaneously optimize the depth-sensing generator, motion trajectory predictor, and spatiotemporal discriminator by utilizing a joint loss function; wherein the joint loss function includes at least a contrastive depth loss for constraining the consistency of depth maps in adjacent frames and an adversarial loss for constraining the realism of the generated video. The final output unit is used to iteratively process the input image until the authenticity score reaches a preset threshold, generating a final output video sequence with high spatiotemporal coherence.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the image-to-video generation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the image-to-video generation method as described in any one of claims 1 to 7.

Images