Video generation method and device, computer equipment and storage medium

Abstract

The invention discloses a video generation method and device, computer equipment and a storage medium, belongs to the technical field of artificial intelligence, and is applied to the financial field or the health medical field. In the tensor algebraic level, high-order tensor decomposition is carried out on an input image by using a transformation tensor product, and the input image is projected to an orthogonal potential subspace. Then, in a feature processing process, weighting the space factor matrix by adopting a local attention window, and combining with a preset mask matrix constraint; a recursive converter with orthogonal constraints is introduced to process a time sequence factor matrix, and it is ensured that the hidden state sequence is stably expressed in long-time dependence modeling. And finally, in a video reconstruction stage, performing up-sampling and reconstruction on the fusion tensor by adopting a tensor completion algorithm based on nuclear norm constraint and a low-rank synthesis strategy. According to the method, the unification of spatial decoupling, time sequence stability and global low-rank performance is realized, so that the generated video reaches a better level in the aspects of spatial resolution, time sequence dynamics and overall visual quality.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, specifically relating to a video generation method, apparatus, computer equipment, and storage medium. Background Technology

[0002] In recent years, with the rapid development of computer vision and artificial intelligence technologies, image-to-video generation technology has become a prominent research hotspot in the fields of computer vision and generative models due to its broad application prospects. In film and television production, video generation technology can quickly transform static images into vivid and coherent video clips, greatly shortening the production cycle and reducing production costs. In game development, it can generate rich and diverse dynamic scenes in real time based on limited image materials, enhancing the immersion and fun of the game. In the field of security monitoring, video generation from static surveillance images helps to more comprehensively reconstruct the event process and complete event analysis and investigation.

[0003] Traditional image-to-video generation methods mainly revolve around Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs). GANs attempt to generate realistic video data through a game between a generator and a discriminator, while VAEs utilize latent space encoding and decoding to construct video generation models. These methods employ an end-to-end training model, directly modeling inter-frame dynamics. Figure 1 This method captures spatial and temporal information from video in a single pass. However, in practical applications, it faces numerous challenges. Because video data contains complex spatial details and temporal dynamics, ensuring both the continuity of spatial details and the plausibility of temporal motion is extremely difficult. During generation, minute changes in spatial details can lead to inconsistencies in temporal motion, and vice versa. This often results in artifacts in the generated video, such as flickering and object distortion, as well as temporal breaks, such as disjointed actions and abrupt scene transitions. These issues severely impact the quality and usability of the generated video, making it difficult to meet the demands of real-world scenarios for high-quality video generation. Summary of the Invention

[0004] The purpose of this application is to provide a video generation method, apparatus, computer equipment, and storage medium, addressing the technical problems that affect the quality and usability of the generated video.

[0005] To address the aforementioned technical problems, this application provides a video generation method, employing the following technical solution: A video generation method, comprising: The input image is received, and a higher-order tensor decomposition is performed on the input image to obtain a third-order tensor of the input image. The third-order tensor consists of a first factor matrix, a second factor matrix, and a third factor matrix. The first and second factor matrices represent spatial features, and the third factor matrix represents temporal dynamic features. The first factor matrix and the second factor matrix are weighted using a preset local attention window to obtain a weighted combination matrix; The weighted combination matrix is ​​masked using a preset mask matrix to obtain a masked weighted matrix; The hidden state sequence is obtained by weighting the third factor matrix using a pre-defined orthogonal weight recursive transformer. The mask weighting matrix and the hidden state sequence are fused to obtain the fused tensor; The fusion tensor is upsampled, and the target video corresponding to the input image is generated based on the upsampling result.

[0006] To address the aforementioned technical problems, this application also provides a video generation apparatus, which employs the following technical solution: A video generation apparatus, comprising: The tensor decomposition module is used to receive the input image and perform high-order tensor decomposition on the input image to obtain the third-order tensor of the input image. The third-order tensor consists of a first factor matrix, a second factor matrix, and a third factor matrix. The first and second factor matrices represent spatial features, and the third factor matrix represents temporal dynamic features. The first weighting module is used to weight the first factor matrix and the second factor matrix using a preset local attention window to obtain a weighted combination matrix; The mask processing module is used to perform mask processing on the weighted combination matrix using a preset mask matrix to obtain the mask weighted matrix; The second weighting module is used to weight the third factor matrix using a preset orthogonal weight recursive transformer to obtain the hidden state sequence. The matrix fusion module is used to fuse the mask weighted matrix and the hidden state sequence to obtain the fused tensor; The video generation module is used to upsample the fused tensor and generate the target video corresponding to the input image based on the upsampling result.

[0007] To address the aforementioned technical problems, this application also provides a computer device that employs the following technical solution: A computer device includes a memory and a processor, the memory storing computer-readable instructions, the processor executing the computer-readable instructions to implement the steps of the video generation method as described in any of the preceding claims.

[0008] To address the aforementioned technical problems, this application also provides a computer-readable storage medium, employing the technical solution described below: A computer-readable storage medium storing computer-readable instructions that, when executed by a processor, implement the steps of the video generation method as described in any one of the preceding descriptions.

[0009] Compared with the prior art, the embodiments of this application have the following main advantages: This application discloses a video generation method, apparatus, computer device, and storage medium, belonging to the field of artificial intelligence technology, and applicable to the financial or healthcare fields. This application effectively ensures the independence of spatial and temporal feature factors by performing high-order tensor decomposition on the input image at the tensor algebra level using transformation tensor product and projecting it to an orthogonal latent subspace. Subsequently, in the feature processing, a local attention window is used to weight the spatial factor matrix, combined with a preset mask matrix constraint, to achieve accurate modeling within local spatial regions. Simultaneously, a recursive transformer with orthogonal constraints is introduced to process the temporal factor matrix, ensuring that the latent state sequence maintains stable expression in long-term dependency modeling, thereby improving the integrity and robustness of temporal features. Finally, in the video reconstruction stage, a tensor completion algorithm based on nuclear norm constraints and a low-rank synthesis strategy are used to upsample and reconstruct the fused tensor, ensuring that the generated video sequence conforms to global low-rank and multi-scale consistency, significantly improving video clarity, inter-frame continuity, and global structural coordination. Overall, this application achieves a balance between spatial decoupling, temporal stability, and global low rank, resulting in videos that exhibit superior spatial resolution, temporal dynamics, and overall visual quality. Attached Figure Description

[0010] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 An exemplary system architecture diagram is shown, in which this application can be applied; Figure 2 A flowchart of one embodiment of the video generation method according to this application is shown; Figure 3 It shows Figure 2 A flowchart of one embodiment of step S201; Figure 4 A schematic diagram of the structure of one embodiment of the video generation apparatus according to this application is shown; Figure 5 It shows Figure 4 A schematic diagram of a embodiment of the tensor decomposition module 401; Figure 6 A schematic diagram of the structure of one embodiment of a computer device according to this application is shown. Detailed Implementation

[0012] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0013] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0014] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0015] like Figure 1 As shown, system architecture 100 may include terminal device 101, network 102, and server 103. Terminal device 101 may be a laptop 1011, tablet 1012, or mobile phone 1013. Network 102 is used as a medium to provide a communication link between terminal device 101 and server 103. Network 102 may include various connection types, such as wired, wireless communication links, or fiber optic cables.

[0016] Users can use terminal device 101 to interact with server 103 via network 102 to receive or send messages, etc. Various communication client applications can be installed on terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0017] Terminal device 101 can be various electronic devices with a display screen and support web browsing. In addition to laptops 1011, tablets 1012, or mobile phones 1013, terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer, and a desktop computer, etc.

[0018] Server 103 can be a server that provides various services, such as a backend server that supports the pages displayed on terminal device 101.

[0019] It should be noted that the video generation method provided in this application embodiment is generally executed by a server / terminal device, and correspondingly, the video generation device is generally located in the server / terminal device.

[0020] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative; the system can have any number of terminal devices, networks, and servers depending on implementation needs.

[0021] Continue to refer to Figure 2 A flowchart of an embodiment of a video generation method according to this application is shown. The video generation method includes the following steps: S201, Receive the input image, perform high-order tensor decomposition on the input image to obtain the third-order tensor of the input image, wherein the third-order tensor consists of a first factor matrix, a second factor matrix and a third factor matrix, the first factor matrix and the second factor matrix represent spatial features, and the third factor matrix represents temporal dynamic features. Specifically, the input raw image is first preprocessed by normalization, mapping pixel values ​​to the range of [-1,1] or [0,1] to eliminate distribution differences caused by different image sources. Next, the normalized image is used to construct an initial tensor representation, typically a three-dimensional structure, where the first and second dimensions correspond to the spatial dimensions of the image, and the third dimension represents the potential temporal dimension. Then, a higher-order tensor decomposition method is introduced. Taking Tucker decomposition as an example, the degrees of freedom are constrained by setting a rank parameter, resulting in a core tensor and three corresponding factor matrices. Alternating least squares is used for iterative optimization during decomposition, and orthogonal constraints are introduced to ensure that the basis vectors of different mode factor matrices are independent. Kernel norm regularization is used to suppress higher-order noise, ensuring low-rank decomposition results. The final first and second factor matrices represent the spatial feature bases of the image in the horizontal and vertical directions, respectively, while the third factor matrix encodes the potential temporal dynamic features of the image.

[0022] Tucker decomposition, a high-order tensor decomposition method, can be viewed as a generalization of singular value decomposition (SVD) to multidimensional tensors. It decomposes a high-order tensor into a product of a smaller core tensor and several factor matrices. The core tensor characterizes the interactions between different modalities, while the factor matrices represent the basis vectors for each modality. In this way, original high-dimensional data can be reduced in dimensionality and compressed while preserving its main features, providing a compact and efficient representation for tasks such as image processing, video generation, and multimodal analysis.

[0023] S202, the first factor matrix and the second factor matrix are weighted using a preset local attention window to obtain a weighted combination matrix; Specifically, the first and second factor matrices are flattened to form an initial spatial feature vector representation. Then, they are combined using the Kronecker product to obtain a joint spatial feature representation matrix, explicitly modeling the correlation between different spatial dimensions. Based on this, the joint spatial features are divided into several non-overlapping or partially overlapping window regions according to a predefined local windowing strategy. The attention mechanism is executed independently within each window. For each window, a query (Q), key (K), and value (V) matrix is ​​constructed, and attention weights are calculated using dot products. These weights are then used to perform a weighted summation of the spatial features within the window, resulting in the weighted features of that local region. To ensure the integrity of texture and structural information, the weighted results of all windows are ultimately combined into a new weighted combination matrix through concatenation or superposition. The entire process emphasizes locality, i.e., limiting the window range to avoid structural blurring caused by long-range pixel dependencies, while simultaneously improving the efficiency and stability of attention computation.

[0024] S203, use a preset mask matrix to mask the weighted combination matrix to obtain a masked weighted matrix; Specifically, firstly, based on the spatial distribution characteristics and local region segmentation results of the input image, a pre-defined mask matrix is ​​generated. This matrix represents whether pixels are connected or adjacent. Next, the weighted combination matrix obtained in the previous step is multiplied element-wise with this mask matrix, thereby eliminating feature connections that do not conform to spatial constraints at the numerical level. After masking, the attention weights of non-adjacent regions are set to zero, ensuring that only features within the local neighborhood can continue to participate in the calculation. After obtaining the initial masking result, a normalization operation is performed to ensure that the weights of the remaining features are within a reasonable range, avoiding numerical overflow or gradient anomalies. Furthermore, to maintain overall dimensionality consistency, the processed masking result is remapped to the same tensor dimension as the weighted combination matrix, generating the final mask weighted matrix. This matrix preserves local weighted features while suppressing invalid dependencies through the masking mechanism.

[0025] S204, the third factor matrix is ​​weighted using a preset orthogonal weight recursive transformer to obtain the hidden state sequence; Specifically, the third factor matrix is ​​first decomposed into temporal feature vectors, which serve as the input sequence for the recursive transformer, and the hidden states of the recursive units are initialized. Then, during the recursive iteration, the input features are weighted using a weight matrix with orthogonal constraints to maintain energy conservation during computation and avoid gradient vanishing or exploding problems. In each iteration, the recursive unit calculates a new hidden state based on the current input and the hidden state from the previous time step. An attention mechanism is introduced during the computation to dynamically adjust the contribution ratio of the input features to the current hidden state. Through this combination of recursion and orthogonal constraints, a sequence of hidden states can be gradually generated, with each hidden state corresponding to a potential temporal frame in the target video. This process continues until the preset target temporal length is reached, thus forming a complete hidden state sequence.

[0026] S205, fuse the mask weighting matrix and the hidden state sequence to obtain the fused tensor; Specifically, firstly, a cross-path attention module is constructed by using the mask weighting matrix as the query vector input and the hidden state sequence as the key-value pair input to establish a correspondence between spatial and temporal features. Next, a dynamic rank-aware mechanism is used to adaptively adjust the number of attention heads based on the tensor's rank, thus flexibly allocating computational resources. Then, in the cross-path attention, the similarity between the query and the key is calculated, and the values ​​are weighted based on this similarity to obtain a weighted result after aligning the temporal and spatial features. This result is then concatenated or element-wise fused with the mask weighting matrix to form a preliminary fused feature. To improve the stability of the fused result, layer normalization and nonlinear transformations are further performed on the fused feature to avoid numerical bias and overfitting. The final output fused tensor has the same dimensionality as the video tensor reconstruction requirements and can be directly used in the upsampling and video generation stages.

[0027] S206, upsample the fused tensor and generate the target video corresponding to the input image based on the upsampling result.

[0028] Specifically, the fused tensor is first input into a low-rank tensor completion module. In this module, a kernel norm constraint is introduced, and the missing information is iteratively completed using the tensor singular value thresholding (t-SVT) algorithm to obtain a low-rank reconstructed tensor. This low-rank reconstructed tensor is then input into a cascaded U-shaped convolutional neural network (U-Net), where convolution, deconvolution, and skip connections are performed sequentially to achieve upsampling of multi-scale features and detail restoration. During this process, a tensor chain format convolution kernel is used to reduce the number of network parameters, while modal multiplication operations at skip connections ensure the alignment of spatial and temporal features. After processing by multiple cascaded U-Net layers, the output video tensor is mapped to a video frame sequence at the target resolution. Finally, inter-frame consistency correction and adaptive histogram matching are performed on the obtained video frame sequence to eliminate potential brightness drift or local differences, thus obtaining a complete target video sequence that meets the overall process requirements from input image to video generation.

[0029] Tensor Singular Value Thresholding (t-SVT) is a high-order data optimization method based on tensor singular value decomposition (t-SVD). It decomposes a tensor into the product of orthogonal tensors and diagonal singular value tensors, and applies a soft threshold to shrink the singular values ​​to achieve a low-rank approximation, thereby suppressing noise or redundant components while preserving the main structural information. Compared with the traditional matrix SVT method, t-SVT, through t-SVD decomposition accelerated by Fourier transform, processes high-dimensional data (such as videos and 3D images) more efficiently, showing significant advantages in data repair, compression, and machine learning tasks. Its core lies in using tensor multimodal correlation to achieve more accurate information preservation and recovery.

[0030] In one specific embodiment, this application discloses an image-to-video generation system based on tensor decomposition and decoupled attention mechanisms. This system achieves highly consistent video synthesis through hierarchical low-rank tensor decomposition and decoupled spatiotemporal attention mechanisms. The core of the system includes a tensor decomposition front-end, a dual-path decoupled attention transformer, and a low-rank synthesis back-end: the front-end uses Tucker decomposition to project the input image onto a latent tensor space, separating spatial appearance from motion dynamics; the dual-path attention module processes the spatial and temporal dimensions respectively through parallel spatial window attention and orthogonal-constrained temporal recursive transformers, and achieves alignment through cross-path attention; the back-end generates video sequences through kernel-norm-constrained low-rank tensor completion and a cascaded U-Net structure. This system strictly guarantees orthogonality and low rank through tensor algebra, while explicitly controlling spatiotemporal fidelity using a decoupling mechanism, outperforming traditional generative adversarial network methods in terms of temporal consistency and visual quality.

[0031] This system is particularly suitable for scenarios requiring high spatiotemporal consistency, such as animate static images in short video marketing and generate background environments during live streaming. By separating the learning process of appearance and motion, users can independently adjust spatial details (such as object textures) and temporal dynamics (such as movement speed), achieving more flexible and controllable video synthesis. Through hierarchical low-rank tensor decomposition and a decoupled spatiotemporal attention mechanism, the system enables the generation of high-quality videos from static images, making it suitable for various applications such as short video editing and digital marketing content creation.

[0032] Further, please see Figure 3 The steps of receiving an input image and performing high-order tensor decomposition on the input image to obtain a third-order tensor of the input image specifically include: S301, Normalize the input image by normalizing the pixel values ​​of the input image to a preset pixel range; S302, Construct an initial tensor representation using the normalized input image; S303, set the rank constraint of each decomposition mode in the higher-order tensor decomposition according to the preset rank parameter, wherein the decomposition mode includes the first decomposition mode, the second decomposition mode and the third decomposition mode, the first decomposition mode corresponds to the first factor matrix, the second decomposition mode corresponds to the second factor matrix, and the third decomposition mode corresponds to the third factor matrix. S304. Based on the rank constraints of each decomposition mode, the initial tensor representation is decomposed into a higher-order tensor to obtain a third-order tensor of the input image.

[0033] In this embodiment, the process of performing high-order tensor decomposition on the input image is achieved by constructing a tensor form of the input image and applying rank constraints. First, the original input image is normalized, mapping pixel values ​​to a preset interval, for example [ [1,1] or [0,1] is used to eliminate the interference caused by brightness differences under different imaging conditions on modeling. Next, an initial tensor representation is constructed using the normalized image and input as a multidimensional array into the tensor decomposition module. Before tensor decomposition, a rank parameter needs to be set according to application requirements and computational resources, i.e., controlling the rank of each decomposition mode. The decomposition modes include the first and second decomposition modes in the spatial dimension, and the third decomposition mode in the temporal dimension. The factor matrices corresponding to the first and second modes are used to capture the spatial structure and appearance features in the image, while the factor matrix corresponding to the third mode is used to model the potential temporal dynamics. By applying low-rank constraints to each mode, data redundancy can be effectively compressed, the compactness of the representation can be improved, and the interpretability of the decomposition results can be enhanced. Based on this, the initial tensor is decomposed using alternating least squares (ALS) or other optimization algorithms, finally obtaining a third-order tensor representation composed of a core tensor and three sets of factor matrices. This third-order tensor not only retains the key feature information of the input image but also decouples spatial features from potential temporal features.

[0034] For example, a fixed input image , represented as It is transformed into a third-order tensor through Tucker decomposition. ,in, Pattern corresponding to the first factor matrix The coding space features, Pattern corresponding to the first factor matrix The coding space features, Pattern corresponding to the third factor matrix The encoding timing dynamics.

[0035] For the factor matrix ( ),satisfy The superscript T stands for matrix transpose. The potential spatial resolution is typically set to the input size. ; The temporal base dimension is the same as the target video length. Linear correlation, For each mode rank, the information compression rate is controlled by learnable parameters.

[0036] Through the above steps, it is possible to achieve decoupled representation of spatial and temporal features while preserving key image features, and generate videos that provide efficient and low-redundancy structured input.

[0037] Furthermore, the rank constraints include orthogonality constraints and low-rank constraints. The steps of performing high-order tensor decomposition on the initial tensor representation based on the rank constraints of each decomposition mode to obtain the third-order tensor of the input image specifically include: The initial tensor representation is iteratively decomposed using the alternating least squares method to obtain the core tensor and the initial factor matrix; During the iterative decomposition process, orthogonal constraints are applied to the initial factor matrices of each decomposition mode; A nuclear norm regularization term is introduced to impose low-rank constraints on the initial factor matrix and the core tensor; Determine whether the core tensor and the initial factor matrix satisfy the iterative convergence condition. If the iterative convergence condition is satisfied, output the first factor matrix, the second factor matrix, and the third factor matrix. The third-order tensor of the input image is reconstructed using the first factor matrix, the second factor matrix, and the third factor matrix.

[0038] In this embodiment, to ensure the stability and expressive power of the high-order tensor decomposition results, a rank control mechanism combining orthogonality and low-rank constraints is introduced into the initial tensor representation decomposition process. Specifically, alternating least squares (ALS) is first used as an optimization strategy to iteratively decompose the initial tensor, gradually obtaining approximate solutions for the core tensor and factor matrices. During the iteration process, orthogonality constraints are applied to the factor matrices of each decomposition mode, requiring that the column vectors of the factor matrices maintain linear independence, thereby avoiding confusion between features of different modes and ensuring the interpretability and numerical stability of the decomposition results. Simultaneously, a kernel norm regularization term is introduced during the decomposition process to impose low-rank constraints on the core tensor and factor matrices, enabling the model to suppress redundant features and noise interference while preserving the main information. This constraint mechanism combining orthogonality and low rank ensures that the representations of each factor matrix in different modes have good separability and sparsity. When the iteration process reaches the convergence condition, the final first, second, and third factor matrices are output. Subsequently, these factor matrices are used to reconstruct the input image using tensors, obtaining the corresponding third-order tensor representation. This decomposition process not only realizes the mapping from image to multimodal feature tensor, but also theoretically guarantees the mathematical rigor and robustness of the results.

[0039] The decomposition process is optimized using alternating least squares (ALS), and the objective function Y is:

[0040] In the formula, the first term is the reconstruction error, and the second term is the nuclear norm regularization. Balancing the two weights, The Frobenius norm, which is the square root of the sum of squares of a matrix or tensor, is often used to measure reconstruction error.

[0041] Through the above steps, an orthogonal and low-rank tensor representation can be obtained, improving feature separation and stability.

[0042] Furthermore, the step of weighting the first factor matrix and the second factor matrix using a preset local attention window to obtain a weighted combination matrix specifically includes: Flatten the first factor matrix and the second factor matrix; Perform a product operation on the flattened first and second factor matrices to generate a joint space matrix; Based on a preset sliding window, the joint space matrix is ​​locally partitioned, and a query matrix, key matrix, and value matrix are constructed within each sliding window; Within each sliding window, attention weights are calculated based on the query matrix, key matrix, and value matrix. Then, the local features within the sliding window are weighted and summed based on the attention weights to obtain the local weighted features. The local weighted features of each sliding window are concatenated to obtain a global weighted combination matrix.

[0043] In this embodiment, to fully utilize the spatial factor matrix information obtained from tensor decomposition of the input image, a weighted combination method based on local attention windows is proposed. First, the first and second factor matrices are flattened, expanding the original multidimensional spatial features into one-dimensional or two-dimensional forms. Then, a product operation, such as the Kronecker product or Hadamard product, is performed on the flattened two sets of factor matrices to generate a joint spatial matrix that comprehensively represents the multidimensional spatial features of the input image. Next, a pre-defined sliding window mechanism is introduced onto the joint spatial matrix, dividing the entire matrix into several local regions, and constructing a query matrix, key matrix, and value matrix within each region. This allows for the establishment of pixel-to-pixel correlation modeling within a local scope. Further, attention weights are calculated based on the query matrix and key matrix within each local window, and these attention weights are used to perform a weighted summation of the features in the value matrix to obtain locally weighted features, thereby highlighting key spatial information within the window. Finally, the weighted features of all local windows are concatenated to form a global weighted combination matrix, achieving the convergence of local features and the reconstruction of global features. This process preserves the spatial locality while taking into account the overall spatial structure.

[0044] For example, the Kronecker product of the spatial factor matrix Perform block self-attention:

[0045] In the formula, The spatial characteristics after flattening; The projection matrix has randomly initialized parameters that can be learned. This is a locally connected mask. It is obtained through a logarithmic mask. Forced spatial locality avoids texture blurring caused by remote dependencies; Kronecker product is used to explicitly model the relationship between spatial dimensions, improving parameter efficiency.

[0046] The above steps can effectively extract local and global spatial features from the input image, improving the accuracy and completeness of spatial information modeling.

[0047] Furthermore, the step of masking the weighted combination matrix using a preset mask matrix to obtain the masked weighted matrix specifically includes: A mask matrix is ​​generated based on the spatial distribution characteristics of the input image, where the mask matrix is ​​used to represent the connectivity between spatial locations; Element-wise multiplication is performed on the weighted combination matrix and the mask matrix to obtain the initial mask result; Elements in the initial mask result that do not satisfy the preset spatial locality constraint are set to zero; Perform a normalization operation on the initial mask result after zeroing; The normalized initial mask result is remapped to a feature representation of the same dimension as the weighted combination matrix to obtain the mask weighted matrix.

[0048] In this embodiment, to further improve the accuracy of spatial feature representation, a mask matrix is ​​introduced to filter and constrain the weighted combination matrix. First, based on the spatial distribution characteristics of the input image, such as the geometric proximity of pixels, edge structure, or region connectivity, a set of mask matrices is generated. The element values ​​of the mask matrix are used to indicate the connectivity between different spatial locations; for example, adjacent pixels are set to 1, and non-adjacent pixels are set to 0, thereby establishing explicit spatial locality constraints. Next, the weighted combination matrix is ​​multiplied element-wise with the mask matrix to obtain an initial mask result. This result can directly eliminate feature responses that do not conform to spatial neighborhood constraints at the matrix level. Subsequently, to avoid the influence of false features caused by remote dependencies or noise interference, elements in the initial mask result that do not satisfy the preset locality constraints are zeroed out, ensuring that only locally consistent parts of the remaining features are retained. To prevent feature distribution imbalance caused by the zeroing operation, the result is normalized to keep the overall numerical range within a reasonable range. Finally, the normalized mask result is mapped back to the same dimension as the original weighted combination matrix, thus obtaining the mask weighted matrix, achieving a unification of feature selection and spatial locality enhancement. Through the above operations, the final generated mask weighted matrix can effectively compress redundant features and highlight important information in local image regions.

[0049] By following the steps above, we can highlight the key features of the local structure of the image, suppress redundant information and noise, and ensure that the spatial feature representation is more accurate.

[0050] Furthermore, the step of weighting the third factor matrix using a pre-defined orthogonal weight recursive transformer to obtain the hidden state sequence specifically includes: The third factor matrix is ​​decomposed into eigenvalues ​​to obtain the initial time series eigenvectors. The initial temporal feature vector is input into the hidden state unit of the recursive transformer, and the hidden state is initialized. During the hidden state update process, an orthogonal constraint is applied to the recursive weight matrix; In each recursive step, attention weights are calculated based on the current hidden state and the initial temporal feature vector of the input, and the hidden state is updated using the attention weights. The updated hidden states are stored sequentially to form a frame-by-frame hidden state sequence; Determine if the sequence has reached the target time length, and output the final hidden state sequence if the target time length has been reached.

[0051] In this embodiment, to extract potential temporal dynamic features from the tensor decomposition results of the input image, an orthogonal weighted recursive transformer is introduced to weight the third factor matrix. Specifically, the third factor matrix is ​​first decomposed into an initial temporal feature vector, which reflects the potential inter-frame variation trend of the video. This temporal feature vector is then input into the hidden state unit of the recursive transformer, and a preset value or a zero vector is assigned to the hidden state during the initialization phase to ensure the stable unfolding of the recursive process. During the update of the hidden state, orthogonal constraints are applied to the recursive weight matrix, forcing it to maintain orthogonality, thereby avoiding gradient explosion or vanishing phenomena during long-sequence training. Furthermore, in each recursive step, attention weights are calculated based on the current hidden state and the input temporal feature vector, and these attention weights are used to weight and update the hidden state, enabling the model to dynamically capture the causal dependencies between different time steps. The updated hidden states are stored sequentially in chronological order, ultimately forming a frame-by-frame sequence of hidden states that can simulate the temporal evolution of the target video. Finally, after the sequence length reaches the target temporal length (such as the target number of video frames), the final hidden state sequence is output.

[0052] For temporal attention paths, the third factor matrix Employing a recursive transformer with orthogonal constraints:

[0053] In the formula, Indicates at time The hidden state vector. Indicates at time The hidden state vector. The output transformation matrix is ​​used to perform a linear transformation on the results of multi-head attention to adjust the dimension and distribution of the feature space, with the following constraints: This refers to orthogonal constraints, which ensure that the linear transformation is an energy-conserving orthogonal mapping, thus preventing gradient explosion or decay of hidden states during iteration. Multi-head attention mechanism is used to address temporal factor matrices. The function extracts and aggregates features relevant to the current context, with the input being the hidden state from the previous step. and time series factor matrix The output is a weighted temporal feature representation. LayerNorm( The ) indicates layer normalization, which is used to normalize the updated hidden states to ensure numerical stability, accelerate convergence, and improve the model's adaptability to different input distributions.

[0054] This formula represents a recursive temporal feature update mechanism with orthogonal constraints. Wherein, Provide time series feature input, Provides historical states, and multi-head attention enables contextual feature fusion. While ensuring energy conservation, LayerNorm guarantees stability, ultimately outputting a new hidden state. .

[0055] The attention weights for each step are calculated as follows:

[0056] In the formula, Indicates at time Attention weight vector, This represents the query weight matrix, whose function is to represent the hidden states. Projecting the query vector onto the query space yields a query vector, which is used to calculate the similarity with the key vector. This represents the key weight matrix, and its function is to combine the time series feature matrix. Projecting onto the key space yields a sequence of key vectors; This represents the dot product of the transpose of all key vectors and the query vector, and the similarity between them is calculated to measure the degree of attention the hidden state pays to temporal features. is the scaling factor, where This represents the dimension of the key vector.

[0057] Through the above steps, the latent temporal dynamics of images can be stably modeled, generating a sequence of hidden states with causal constraints, providing temporal drive for video synthesis.

[0058] The unique features of the above decomposition are: maintaining the integrity of the tensor structure by transforming the tensor product, avoiding the spatial-temporal confusion caused by traditional matricization; and ensuring the independence of the basis vectors of each dimension by the orthogonal constraint of the factor matrix.

[0059] Furthermore, the step of upsampling the fusion tensor and generating the target video corresponding to the input image based on the upsampling result specifically includes: The fused tensor is iteratively optimized using a tensor singular value thresholding algorithm to obtain a low-rank reconstructed tensor; The low-rank reconstruction tensor is input into a cascaded U-shaped convolutional neural network built on the tensor chain format, and layer-by-layer convolution and deconvolution operations are performed to obtain multi-scale upsampling features. Modal product operations are performed on multi-scale upsampled features in a cascaded U-shaped convolutional neural network to obtain the modal product operation results; Map the modal product operation result to a video frame sequence at the target resolution; Perform inter-frame consistency correction and brightness matching on the video frame sequence, and output the target video corresponding to the input image.

[0060] In this embodiment, to transform the fused tensor into a complete target video, a tensor singular value thresholding algorithm is first used to iteratively optimize the fused tensor. This method gradually compresses the singular values ​​of the tensor, weakening redundant information and highlighting the main components, thereby obtaining a low-rank reconstructed tensor that ensures a more compact data representation while retaining key information. Subsequently, the low-rank reconstructed tensor is input into a cascaded U-shaped convolutional neural network (U-Net) built based on the tensor chain format. This network extracts and restores features from the tensor through layer-by-layer convolution and deconvolution operations, forming a multi-scale upsampled feature representation. Through this multi-scale mechanism, detailed information at different spatial scales in the target video can be effectively recovered. Simultaneously, modal product operations are further applied to the multi-scale features in the cascaded U-Net, deeply fusing spatial and temporal modal features, so that the generated video frames can achieve a balance between temporal consistency and spatial sharpness. Then, the modal product operation results are mapped to a video frame sequence that conforms to the target resolution, realizing the restoration process from low-dimensional latent representation to real video. Finally, to improve the visual stability of the generated video, inter-frame consistency correction and brightness matching operations are performed on the video frame sequence to reduce discontinuities such as flickering and color shift, ensuring smoothness and naturalness of the video during dynamic playback.

[0061] By following the steps above, target videos with consistent resolution, smooth temporal sequence, and rich detail can be generated, significantly improving the visual quality and viewing experience of the videos.

[0062] In the above embodiments, this application discloses a video generation method, belonging to the field of artificial intelligence technology, and applicable to the financial or healthcare fields. This application effectively ensures the independence of spatial and temporal feature factors by using a transformation tensor product to perform high-order tensor decomposition on the input image at the tensor algebra level and projecting it onto an orthogonal latent subspace. Subsequently, in the feature processing, a local attention window is used to weight the spatial factor matrix, combined with a preset mask matrix constraint, to achieve accurate modeling within local spatial regions. Simultaneously, a recursive transformer with orthogonal constraints is introduced to process the temporal factor matrix, ensuring that the latent state sequence maintains stable expression in long-term dependency modeling, thereby improving the integrity and robustness of temporal features. Finally, in the video reconstruction stage, a tensor completion algorithm based on nuclear norm constraints and a low-rank synthesis strategy are used to upsample and reconstruct the fused tensor, ensuring that the generated video sequence conforms to global low-rank and multi-scale consistency, significantly improving video clarity, inter-frame continuity, and global structural coordination. Overall, this application achieves a balance between spatial decoupling, temporal stability, and global low rank, resulting in videos that exhibit superior spatial resolution, temporal dynamics, and overall visual quality.

[0063] In this embodiment, the video generation method runs on an electronic device (e.g., Figure 1 The server shown can receive instructions or acquire data via wired or wireless connection. It should be noted that the aforementioned wireless connection methods may include, but are not limited to, 3G / 4G connections, WiFi connections, Bluetooth connections, WiMAX connections, Zigbee connections, UWB (ultra-wideband) connections, and other currently known or future wireless connection methods.

[0064] It should be emphasized that, in order to further ensure the privacy and security of the above-mentioned input image information, the input image information can also be stored in a blockchain node.

[0065] The blockchain referred to in this application is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0066] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0067] Foundational technologies in artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0068] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware through computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0069] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0070] Further reference Figure 4 As a response to the above Figure 2 To implement the method shown, this application provides an embodiment of a video generation apparatus, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0071] like Figure 4 As shown, the video generation device 400 described in this embodiment includes: Tensor decomposition module 401 is used to receive the input image and perform high-order tensor decomposition based on the input image to obtain the third-order tensor of the input image. The third-order tensor consists of a first factor matrix, a second factor matrix, and a third factor matrix. The first factor matrix and the second factor matrix represent spatial features, and the third factor matrix represents temporal dynamic features. The first weighting module 402 is used to weight the first factor matrix and the second factor matrix using a preset local attention window to obtain a weighted combination matrix; The mask processing module 403 is used to perform mask processing on the weighted combination matrix using a preset mask matrix to obtain a mask weighted matrix; The second weighting module 404 is used to weight the third factor matrix using a preset orthogonal weight recursive transformer to obtain the hidden state sequence. The matrix fusion module 405 is used to fuse the mask weighting matrix and the hidden state sequence to obtain a fused tensor; The video generation module 406 is used to upsample the fused tensor and generate the target video corresponding to the input image based on the upsampling result.

[0072] Further, please see Figure 5 The tensor decomposition module 401 specifically includes: The normalization processing unit 501 is used to normalize the input image and normalize the pixel values ​​of the input image to a preset pixel range. Tensor construction unit 502 is used to construct an initial tensor representation using the normalized input image; The rank parameter setting unit 503 is used to set the rank constraints of each decomposition mode in the higher-order tensor decomposition according to the preset rank parameter. The decomposition mode includes a first decomposition mode, a second decomposition mode and a third decomposition mode. The first decomposition mode corresponds to the first factor matrix, the second decomposition mode corresponds to the second factor matrix and the third decomposition mode corresponds to the third factor matrix. The higher-order tensor decomposition unit 504 is used to perform higher-order tensor decomposition on the initial tensor representation based on the rank constraints of each decomposition mode to obtain the third-order tensor of the input image.

[0073] Furthermore, the rank constraint includes orthogonality constraint and low-rank constraint, and the higher-order tensor decomposition unit 504 specifically includes: Iterative decomposition subunits are used to iteratively decompose the initial tensor representation using the alternating least squares method to obtain the core tensor and the initial factor matrix; Orthogonal constraint sub-units are used to apply orthogonal constraints to the initial factor matrices of each decomposition mode during the iterative decomposition process; Low-rank constraint subunits are used to introduce nuclear norm regularization terms to impose low-rank constraints on the initial factor matrix and core tensor. The convergence condition judgment subunit is used to determine whether the core tensor and the initial factor matrix satisfy the iterative convergence condition. When the iterative convergence condition is satisfied, the first factor matrix, the second factor matrix, and the third factor matrix are output. Tensor reconstruction subunit is used to reconstruct the third-order tensor of the input image using the first factor matrix, the second factor matrix, and the third factor matrix.

[0074] Furthermore, the first weighting module 402 specifically includes: The flattening unit is used to flatten the first factor matrix and the second factor matrix; The product operation unit is used to perform product operations on the flattened first factor matrix and second factor matrix to generate a joint space matrix. A window sliding unit is used to perform local partitioning on the joint space matrix based on a preset sliding window, and to construct a query matrix, a key matrix, and a value matrix within each sliding window; The attention weight calculation unit is used to calculate the attention weight based on the query matrix, key matrix, and value matrix within each sliding window, and to perform a weighted summation of the local features in the sliding window based on the attention weight to obtain the local weighted features; The feature concatenation unit is used to concatenate the local weighted features of each sliding window to obtain a global weighted combination matrix.

[0075] Furthermore, the mask processing module 403 specifically includes: The mask matrix generation unit is used to generate a mask matrix based on the spatial distribution characteristics of the input image, wherein the mask matrix is ​​used to represent the connectivity between spatial locations; The mask processing unit is used to perform element-wise multiplication on the weighted combination matrix and the mask matrix to obtain the initial mask result. The zeroing processing unit is used to zero out elements in the initial mask result that do not satisfy the preset spatial locality constraint. The normalization operation unit is used to perform normalization operations on the initial mask result after zeroing. The result mapping unit is used to remap the normalized initial mask result into a feature representation of the same dimension as the weighted combination matrix, thus obtaining the mask weighted matrix.

[0076] Furthermore, the second weighting module 404 specifically includes: The eigenvalue decomposition unit is used to perform eigenvalue decomposition on the third factor matrix to obtain the initial time series eigenvector. The hidden state initialization unit is used to input the initial temporal feature vector into the hidden state unit of the recursive transformer and initialize the hidden state. Recursive orthogonal constraint unit, used to apply orthogonal constraints to the recursive weight matrix during the hidden state update process; The weighted update unit is used to calculate the attention weights based on the current hidden state and the initial temporal feature vector of the input in each recursive step, and to perform a weighted update of the hidden state using the attention weights. The hidden state storage unit is used to store the updated hidden states sequentially, forming a frame-by-frame hidden state sequence. The hidden state sequence acquisition unit is used to determine whether the sequence has reached the target time length, and outputs the final hidden state sequence after reaching the target time length.

[0077] Furthermore, the video generation module 406 specifically includes: The low-rank reconstruction unit is used to iteratively optimize the fused tensor using the tensor singular value thresholding algorithm to obtain the low-rank reconstruction tensor; The upsampling unit is used to input the low-rank reconstructed tensor into a cascaded U-shaped convolutional neural network built on the tensor chain format, and perform layer-by-layer convolution and deconvolution operations to obtain multi-scale upsampling features. The modal product unit is used to perform modal product operations on multi-scale upsampled features in a cascaded U-shaped convolutional neural network to obtain the modal product operation results; The video frame mapping unit is used to map the modal product operation result into a video frame sequence with a target resolution. The video frame verification unit is used to perform inter-frame consistency correction and brightness matching on the video frame sequence, and output the target video corresponding to the input image.

[0078] In the above embodiments, this application discloses a video generation device, belonging to the field of artificial intelligence technology, and applicable to the financial or healthcare fields. This application effectively ensures the independence of spatial and temporal feature factors by using a transformation tensor product to perform high-order tensor decomposition on the input image at the tensor algebra level and projecting it onto an orthogonal latent subspace. Subsequently, in the feature processing, a local attention window is used to weight the spatial factor matrix, combined with a preset mask matrix constraint, to achieve accurate modeling within local spatial regions. Simultaneously, a recursive transformer with orthogonal constraints is introduced to process the temporal factor matrix, ensuring that the latent state sequence maintains stable expression in long-term dependency modeling, thereby improving the integrity and robustness of temporal features. Finally, in the video reconstruction stage, a tensor completion algorithm based on nuclear norm constraints and a low-rank synthesis strategy are used to upsample and reconstruct the fused tensor, ensuring that the generated video sequence conforms to global low-rank and multi-scale consistency, significantly improving video clarity, inter-frame continuity, and global structural coordination. Overall, this application achieves a balance between spatial decoupling, temporal stability, and global low rank, resulting in videos that exhibit superior spatial resolution, temporal dynamics, and overall visual quality.

[0079] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 6 , Figure 6 This is a basic structural block diagram of the computer device in this embodiment.

[0080] The computer device 6 includes a memory 61, a processor 62, and a network interface 63 that are interconnected via a system bus. It should be noted that only the computer device 6 with memory 61, processor 62, and network interface 63 is shown in the figure; however, it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0081] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0082] The memory 61 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 61 may be an internal storage unit of the computer device 6, such as the hard disk or memory of the computer device 6. In other embodiments, the memory 61 may also be an external storage device of the computer device 6, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 6. Of course, the memory 61 may also include both the internal storage unit and its external storage device of the computer device 6. In this embodiment, the memory 61 is typically used to store the operating system and various application software installed on the computer device 6, such as computer-readable instructions for video generation methods. In addition, the memory 61 can also be used to temporarily store various types of data that have been output or will be output.

[0083] In some embodiments, the processor 62 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 62 is typically used to control the overall operation of the computer device 6. In this embodiment, the processor 62 is used to execute computer-readable instructions stored in the memory 61 or to process data, for example, to execute computer-readable instructions for the video generation method.

[0084] The network interface 63 may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 6 and other electronic devices.

[0085] This application also provides an embodiment, namely, a computer device including a memory and a processor. The memory stores computer-readable instructions, and the processor, when executing the computer-readable instructions, implements the steps of the video generation method described above, that is, it implements: A video generation method, comprising: The input image is received, and a higher-order tensor decomposition is performed on the input image to obtain a third-order tensor of the input image. The third-order tensor consists of a first factor matrix, a second factor matrix, and a third factor matrix. The first and second factor matrices represent spatial features, and the third factor matrix represents temporal dynamic features. The first factor matrix and the second factor matrix are weighted using a preset local attention window to obtain a weighted combination matrix; The weighted combination matrix is ​​masked using a preset mask matrix to obtain a masked weighted matrix; The hidden state sequence is obtained by weighting the third factor matrix using a pre-defined orthogonal weight recursive transformer. The mask weighting matrix and the hidden state sequence are fused to obtain the fused tensor; The fusion tensor is upsampled, and the target video corresponding to the input image is generated based on the upsampling result.

[0086] This application also provides another embodiment, namely, a computer-readable storage medium storing computer-readable instructions that can be executed by at least one processor to cause the at least one processor to perform the steps of the video generation method described above, i.e., to implement: A video generation method, comprising: The input image is received, and a higher-order tensor decomposition is performed on the input image to obtain a third-order tensor of the input image. The third-order tensor consists of a first factor matrix, a second factor matrix, and a third factor matrix. The first and second factor matrices represent spatial features, and the third factor matrix represents temporal dynamic features. The first factor matrix and the second factor matrix are weighted using a preset local attention window to obtain a weighted combination matrix; The weighted combination matrix is ​​masked using a preset mask matrix to obtain a masked weighted matrix; The hidden state sequence is obtained by weighting the third factor matrix using a pre-defined orthogonal weight recursive transformer. The mask weighting matrix and the hidden state sequence are fused to obtain the fused tensor; The fusion tensor is upsampled, and the target video corresponding to the input image is generated based on the upsampling result.

[0087] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0088] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0089] It should be noted that the software tools or components not belonging to this company that appear in the various embodiments of this application are merely illustrative examples and do not represent actual use.

[0090] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. A video generation method, characterized in that, include: The input image is received, and a higher-order tensor decomposition is performed on the input image to obtain a third-order tensor of the input image. The third-order tensor consists of a first factor matrix, a second factor matrix, and a third factor matrix. The first factor matrix and the second factor matrix represent spatial features, and the third factor matrix represents temporal dynamic features. The first factor matrix and the second factor matrix are weighted using a preset local attention window to obtain a weighted combination matrix; The weighted combination matrix is ​​masked using a preset mask matrix to obtain a masked weighted matrix; The third factor matrix is ​​weighted using a preset orthogonal weight recursive transformer to obtain the hidden state sequence; The mask weighting matrix and the hidden state sequence are fused to obtain a fused tensor; The fused tensor is upsampled, and the target video corresponding to the input image is generated based on the upsampling result.

2. The video generation method as described in claim 1, characterized in that, The step of receiving the input image and performing high-order tensor decomposition based on the input image to obtain a third-order tensor of the input image specifically includes: The input image is normalized to normalize the pixel values ​​of the input image to a preset pixel range; An initial tensor representation is constructed using the normalized input image; The rank constraints of each decomposition mode in the higher-order tensor decomposition are set according to the preset rank parameter. The decomposition mode includes a first decomposition mode, a second decomposition mode and a third decomposition mode. The first decomposition mode corresponds to the first factor matrix, the second decomposition mode corresponds to the second factor matrix, and the third decomposition mode corresponds to the third factor matrix. Based on the rank constraints of each of the decomposition modes, the initial tensor representation is decomposed into a higher-order tensor to obtain a third-order tensor of the input image.

3. The video generation method as described in claim 2, characterized in that, The rank constraints include orthogonality constraints and low-rank constraints. The step of performing high-order tensor decomposition on the initial tensor representation based on the rank constraints of each decomposition mode to obtain the third-order tensor of the input image specifically includes: The initial tensor representation is iteratively decomposed using the alternating least squares method to obtain the core tensor and the initial factor matrix; During the iterative decomposition process, orthogonal constraints are applied to the initial factor matrices of each decomposition mode; A nuclear norm regularization term is introduced to impose low-rank constraints on the initial factor matrix and the core tensor. Determine whether the core tensor and the initial factor matrix satisfy the iterative convergence condition, and output the first factor matrix, the second factor matrix and the third factor matrix when the iterative convergence condition is satisfied; The third-order tensor of the input image is reconstructed using the first factor matrix, the second factor matrix, and the third factor matrix.

4. The video generation method as described in claim 1, characterized in that, The step of weighting the first factor matrix and the second factor matrix using a preset local attention window to obtain a weighted combination matrix specifically includes: Flatten the first factor matrix and the second factor matrix; Perform a product operation on the flattened first factor matrix and the second factor matrix to generate a joint space matrix; Based on a preset sliding window, the joint space matrix is ​​locally partitioned, and a query matrix, a key matrix, and a value matrix are constructed within each sliding window. Within each sliding window, attention weights are calculated based on the query matrix, the key matrix, and the value matrix. Then, local features within the sliding window are weighted and summed based on the attention weights to obtain local weighted features. The local weighted features of each sliding window are concatenated to obtain a global weighted combination matrix.

5. The video generation method as described in claim 4, characterized in that, The step of masking the weighted combination matrix using a preset mask matrix to obtain a masked weighted matrix specifically includes: The mask matrix is ​​generated based on the spatial distribution characteristics of the input image, wherein the mask matrix is ​​used to represent the connectivity between spatial locations; Element-wise multiplication is performed on the weighted combination matrix and the mask matrix to obtain the initial mask result; Elements in the initial mask result that do not satisfy the preset spatial locality constraint are set to zero; The initial mask result after being zeroed out is normalized. The normalized initial mask result is remapped to a feature representation of the same dimension as the weighted combination matrix to obtain the mask weighted matrix.

6. The video generation method as described in claim 1, characterized in that, The step of weighting the third factor matrix using a preset orthogonal weight recursive transformer to obtain the hidden state sequence specifically includes: The third factor matrix is ​​subjected to eigenvalue decomposition to obtain the initial time series feature vector; The initial temporal feature vector is input into the hidden state unit of the recursive transformer, and the hidden state is initialized. During the hidden state update process, an orthogonal constraint is applied to the recursive weight matrix; In each recursive step, attention weights are calculated based on the current hidden state and the initial temporal feature vector of the input, and the hidden state is updated using the attention weights. The updated hidden states are stored sequentially to form a frame-by-frame hidden state sequence; Determine whether the sequence has reached the target time length, and output the final hidden state sequence after reaching the target time length.

7. The video generation method as described in claim 1, characterized in that, The step of upsampling the fused tensor and generating the target video corresponding to the input image based on the upsampling result specifically includes: The fused tensor is iteratively optimized using a tensor singular value thresholding algorithm to obtain a low-rank reconstructed tensor. The low-rank reconstructed tensor is input into a cascaded U-shaped convolutional neural network built on the tensor chain format, and layer-by-layer convolution and deconvolution operations are performed to obtain multi-scale upsampling features. In the cascaded U-shaped convolutional neural network, modal multiplication is performed on the multi-scale upsampled features to obtain the modal multiplication result; The modal product operation result is mapped to a video frame sequence at the target resolution. The video frame sequence is subjected to inter-frame consistency correction and brightness matching, and the target video corresponding to the input image is output.

8. A video generation apparatus, characterized in that, include: The tensor decomposition module is used to receive an input image and perform high-order tensor decomposition on the input image to obtain a third-order tensor of the input image. The third-order tensor consists of a first factor matrix, a second factor matrix, and a third factor matrix. The first factor matrix and the second factor matrix represent spatial features, and the third factor matrix represents temporal dynamic features. The first weighting module is used to weight the first factor matrix and the second factor matrix using a preset local attention window to obtain a weighted combination matrix; The masking module is used to perform masking processing on the weighted combination matrix using a preset masking matrix to obtain a masked weighted matrix; The second weighting module is used to weight the third factor matrix using a preset orthogonal weight recursive transformer to obtain the hidden state sequence. The matrix fusion module is used to fuse the mask weighting matrix and the hidden state sequence to obtain a fused tensor; The video generation module is used to upsample the fused tensor and generate the target video corresponding to the input image based on the upsampling result.

9. A computer device, characterized in that, The device includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the video generation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the video generation method as described in any one of claims 1 to 7.

Images