Video generation model training method, video generation method, device and equipment

Abstract

The present application provides a training method, a video generation method, an apparatus and a device for a video generation model, and relates to artificial intelligence technology. The method includes: for training data consisting of static images containing different medical parts, determining a selection coefficient matrix representing the type of different medical parts by image classification; fusing the selection coefficient matrix with the initialized weight matrix to obtain a motion template matrix; inputting the static image into a VAE encoder, outputting a coded potential representation through the VAE encoder; inputting the coded potential representation into a UNET network, and adding the motion template matrix to the temporal attention layer of the UNET network; inputting the output result of the UNET network with the motion template matrix into a VAE decoder, and generating the current video through the VAE decoder; calculating the current loss based on the real video corresponding to the static image and the current video, and optimizing the parameters of the temporal attention layer and weight matrix of the UNET network according to the current loss. The present application can improve the authenticity of the generated medical part animation.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a video generation model training method, video generation method, device and equipment. Background Art

[0002] High-quality animations of blood vessels and internal organs play a vital role in medical education, clinical diagnosis, and science popularization. However, existing animation production technologies have significant shortcomings in simulating the dynamic behavior of medical parts such as blood vessels and internal organs. Summary of the Invention

[0003] The present application provides a video generation model training method, video generation method, device and equipment to solve the problem that the existing technology has significant deficiencies in simulating the dynamic behavior of medical parts such as blood vessels and internal organs.

[0004] In a first aspect, the present application provides a method for training a video generation model, wherein the video generation model includes a pre-trained VAE encoder, a VAE decoder, and a UNET network, wherein the UNET network includes multiple groups of attention layers consisting of temporal attention layers and spatial attention layers, and the method includes:

[0005] For training data consisting of static images containing different medical parts, determining a selection coefficient matrix representing the types of different medical parts in the static images by image classification;

[0006] fusing the selection coefficient matrix with a pre-initialized weight matrix to obtain a motion template matrix, wherein the weight matrix is ​​used to characterize the motion trend information of the different medical parts;

[0007] Input the static image into the VAE encoder, and output the encoded latent representation through the VAE encoder;

[0008] Inputting the encoded latent representation into the UNET network and adding the motion template matrix into the temporal attention layer of the UNET network;

[0009] Inputting the output result of the UNET network to which the motion template matrix is ​​added into the VAE decoder, and generating the current video through the VAE decoder;

[0010] A current loss is calculated based on a real video corresponding to the static image and the current video, and parameters of the temporal attention layer and the weight matrix of the UNET network are optimized according to the current loss, wherein the parameters of the spatial attention layer of the UNET network are frozen during training.

[0011] In a second aspect, the present application provides a video generation method, comprising:

[0012] Obtain a video generation model and a trained weight matrix, wherein the video generation model is trained according to the video generation model training method described in any of the embodiments of the present application, and the video generation model includes a VAE encoder, a VAE decoder, and a UNET network;

[0013] performing image classification on a target image containing different medical parts to obtain a target selection coefficient matrix representing the types of the different medical parts in the target image;

[0014] fusing the target selection coefficient matrix with the trained weight matrix to obtain a target motion template matrix, wherein the weight matrix is ​​used to characterize the motion trend information of the different medical parts;

[0015] Input the target image into the VAE encoder, and output the target encoded latent representation through the VAE encoder;

[0016] Inputting the target encoding potential representation into the UNET network, and adding the target motion template matrix into the temporal attention layer of the UNET network;

[0017] The output result of the UNET network to which the motion template matrix is ​​added is input into the VAE decoder, and the target video is generated by the VAE decoder.

[0018] In a third aspect, the present application provides a training device for a video generation model, wherein the video generation model includes a pre-trained VAE encoder, a VAE decoder, and a UNET network, wherein the UNET network includes multiple groups of attention layers consisting of temporal attention layers and spatial attention layers, and the device includes:

[0019] An image classification module is configured to determine, for training data consisting of static images containing different medical parts, a selection coefficient matrix representing the types of the different medical parts in the static images by image classification;

[0020] a motion template selection module, configured to fuse the selection coefficient matrix with a pre-initialized weight matrix to obtain a motion template matrix, wherein the weight matrix is ​​used to characterize motion trend information of the different medical parts;

[0021] A first input module is configured to input the static image into the VAE encoder and output an encoded latent representation through the VAE encoder;

[0022] A motion template adding module, configured to input the encoded potential representation into the UNET network and add the motion template matrix into the temporal attention layer of the UNET network;

[0023] A second input module is used to input the output result of the UNET network added with the motion template matrix into the VAE decoder, and generate the current video through the VAE decoder;

[0024] An optimization training module is used to calculate a current loss based on a real video corresponding to the static image and the current video, and optimize the parameters of the temporal attention layer and the weight matrix of the UNET network according to the current loss, wherein the parameters of the spatial attention layer of the UNET network are frozen during the training process.

[0025] In a fourth aspect, the present application provides a video generation device, comprising:

[0026] A model acquisition module, configured to acquire a video generation model and a trained weight matrix, wherein the video generation model is trained by the video generation model training device described in any one of the embodiments of the present application, and the video generation model includes a VAE encoder, a VAE decoder, and a UNET network;

[0027] a target selection coefficient matrix determination module, configured to classify target images containing different medical parts to obtain a target selection coefficient matrix representing the types of the different medical parts in the target image;

[0028] a target motion template matrix determination module, configured to fuse the target selection coefficient matrix with the trained weight matrix to obtain a target motion template matrix, wherein the weight matrix is ​​used to characterize motion trend information of the different medical parts;

[0029] a target encoding potential representation determination module, configured to input the target image into the VAE encoder and output the target encoding potential representation through the VAE encoder;

[0030] A target motion template adding module is used to input the target encoding potential representation into the UNET network and add the target motion template matrix to the temporal attention layer of the UNET network;

[0031] The target video generation module is used to input the output result of the UNET network with the motion template matrix added into the VAE decoder, and generate the target video through the VAE decoder.

[0032] In a fifth aspect, the present application also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the program, it implements the video generation model training method and video generation method as described in any one of the embodiments of the present application.

[0033] In a sixth aspect, the present application also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, it implements the video generation model training method and video generation method as described in any one of the embodiments of the present application.

[0034] In a seventh aspect, the present application also provides a computer program product, comprising a computer program, which, when executed by a processor, implements the video generation model training method and video generation method as described in any one of the embodiments of the present application.

[0035] The training method, video generation method, apparatus, and device of the video generation model provided in this application classify static images in the training data to determine a selection coefficient matrix representing the types of different medical parts in the static image, then fuse the selection coefficient matrix with a pre-initialized weight matrix to obtain a motion template matrix, and then add the motion template matrix to the temporal attention layer of the UNET network. During the training process, the parameters of the spatial attention layer of the UNET network are frozen, and the parameters of the temporal attention layer and weight matrix of the UNET network are optimized. Thus, the weight matrix is ​​used to learn the motion trend information representing different medical parts, so that the trained video generation model can generate more realistic animated videos of medical parts based on the weight matrix. The animated videos are more consistent with the natural motion trends of the medical parts, thereby improving the accuracy and scientificity of the generated animation. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0037] Figure 1 A flowchart of a method for training a video generation model provided in an embodiment of the present application;

[0038] Figure 2a A schematic diagram of semantic segmentation in the training method provided in an embodiment of the present application;

[0039] Figure 2b A schematic diagram of a medical part after semantic segmentation in the training method provided in an embodiment of the present application;

[0040] Figure 3 A schematic diagram of another video generation model training method provided in an embodiment of the present application;

[0041] Figure 4 A flowchart of a video generation method provided in an embodiment of the present application;

[0042] Figure 5 A schematic diagram of an animation production process provided in an embodiment of the present application;

[0043] Figure 6 A schematic diagram of the structure of a video generation model training device provided in an embodiment of the present application;

[0044] Figure 7 A schematic diagram of the structure of a video generation device provided in an embodiment of the present application;

[0045] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0046] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0047] The present application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are intended only to illustrate the present application and are not intended to limit the present application. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions of the present application, not all of the structures.

[0048] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance. The acquisition, storage, use, processing, etc. of data in the technical solution of this application comply with the relevant provisions of national laws and regulations. It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of the implementation of the technical solution of this application, but it does not mean that the applicant has or must use the relevant content of the solution.

[0049] Figure 1A flowchart of a method for training a video generation model provided in an embodiment of the present application. This embodiment is applicable to training a video generation model so as to generate animated videos of medical parts using the trained video generation model. The method can be performed by a video generation model training device, which can be implemented in software and / or hardware, and is preferably configured in an electronic device, such as a computer or server.

[0050] The video generation model in the embodiment of the present application includes a pre-trained VAE encoder (Variational Auto-Encoder), a VAE decoder (Variational Auto-Decoder) and a UNET network (a symmetrical network structure). The UNET network includes multiple attention layers consisting of a temporal attention layer and a spatial attention layer. Figure 1 As shown, the method specifically includes:

[0051] S101 . For training data consisting of static images containing different medical parts, determine a selection coefficient matrix representing the types of different medical parts in the static images through image classification.

[0052] The present application does not impose any restrictions on the number of medical parts contained in a static image. By image classification, the various medical parts contained in the static image can be identified and their types determined. Types include, for example, tubular, visceral, cellular, or other. Tubular medical parts may include blood vessels, trachea, and intestines, visceral medical parts may include the heart, liver, spleen, or kidneys, and cellular medical parts may include platelets or bacteria. By classifying and identifying the different medical parts contained in the static image, their types are determined, thereby obtaining a selection coefficient matrix.

[0053] In one embodiment, determining a selection coefficient matrix representing the types of different medical parts in a static image through image classification includes: performing semantic segmentation on the static image to obtain multiple medical part images; and performing image classification on the multiple medical part images using an image classification model to determine a selection coefficient matrix representing the types of the multiple medical parts.

[0054] Among them, any semantic segmentation model in the prior art can be used to perform semantic segmentation on the static image, and the obtained medical part image can contain only one medical part or multiple medical parts. The image classification model is then used to identify and classify the medical part images obtained by segmentation to determine the type to which they belong, thereby obtaining a selection coefficient matrix. Exemplarily, the selection coefficient matrix can be a one-dimensional matrix, such as [0.2, 0.2, 0.5, 0.1], which contains four dimensions, each dimension representing four predefined types, such as tubular, visceral, cellular or other. The number on each dimension represents the confidence or probability that the medical part image belongs to that type. If a static image contains a cell image and the cell image is segmented through semantic segmentation, then the selection coefficient matrix obtained by the image classification model can be [0.01, 0.01, 0.9, 0.08], where the value of the third dimension, 0.9, is the largest, indicating that the probability that the image belongs to a cell is the highest; if the segmented image contains both cells and internal organs, then the obtained one-dimensional matrix can be [0.01, 0.4, 0.4, 0.09], where the values ​​of the second and third dimensions are both 0.4, and are the largest, indicating that the probability that the image belongs to both cells and internal organs is the highest.

[0055] Figure 2a A schematic diagram of semantic segmentation in the training method provided in an embodiment of the present application. Figure 2b Schematic diagram of medical parts after semantic segmentation in the training method provided in the embodiment of the present application. As shown in the figure, semantic segmentation can be used to determine the images of different medical parts contained in a static image.

[0056] S102 , fusing the selection coefficient matrix with the pre-initialized weight matrix to obtain a motion template matrix, wherein the weight matrix is ​​used to represent motion trend information of different medical parts.

[0057] S103: Input the static image into the VAE encoder, and output the encoded latent representation through the VAE encoder.

[0058] Specifically, a static image can be input into a VAE encoder to obtain an image latent representation. Noise is then added to the feature map of the static image obtained by the VAE encoder to obtain a noise latent representation. The image latent representation and the noise latent representation are then concatenated to obtain the encoded latent representation. The number of frames in the noise latent representation can be the same as the number of frames in the current video generated by the VAE decoder. For details about the VAE encoder, please refer to the prior art and will not be repeated here.

[0059] S104. Input the encoded potential representation into the UNET network, and add the motion template matrix to the temporal attention layer of the UNET network.

[0060] S105. Input the output result of the UNET network with the motion template matrix added into the VAE decoder, and generate the current video through the VAE decoder.

[0061] S106. Calculate the current loss based on the real video corresponding to the static image and the current video, and optimize the parameters of the temporal attention layer and the weight matrix of the UNET network according to the current loss, wherein the parameters of the spatial attention layer of the UNET network are frozen during the training process.

[0062] Different medical parts exhibit distinct motion trends. For example, blood flowing through blood vessels exhibits distinct motion trends compared to the pulsation of internal organs. High-quality animations of blood vessels and internal organs play a crucial role in medical education, clinical diagnosis, and scientific outreach. However, existing animation techniques have significant shortcomings in simulating the dynamic behavior of blood vessels and internal organs.

[0063] In response to this problem, an embodiment of the present application introduces a weight matrix that can characterize the motion trend information of different medical parts during the training process of the video generation model. Specifically, the weight matrix is ​​first randomly initialized, and then the selection coefficient matrix is ​​fused with the pre-initialized weight matrix. Preferably, the selection coefficient matrix can be multiplied by the pre-initialized weight matrix to obtain a motion template matrix. During training, a static image is input into the VAE encoder, the encoded potential representation is output by the VAE encoder, and the encoded potential representation is then input into the UNET network, and the motion template matrix is ​​added to the temporal attention layer of the UNET network. The UNET network with the motion template matrix added processes the input encoded potential representation, and its output result is input into the VAE decoder, and the current video is generated by the VAE decoder. The current loss is calculated based on the real video corresponding to the static image and the current video, and the parameters of the temporal attention layer and weight matrix of the UNET network are optimized according to the current loss, wherein the parameters of the spatial attention layer of the UNET network are frozen during the training process. The video generation model and weight matrix are trained according to the above method until the convergence conditions are met, and the parameters of the temporal attention layer and weight matrix can be determined. Among them, the VAE encoder and VAE decoder can be obtained through pre-training, and can also be optimized during the training process of the embodiment of the present application.

[0064] Furthermore, the motion template matrix can be added to the UNET network's temporal attention layer via a cross-attention layer. Specifically, the selection coefficient matrix is ​​first multiplied by the weight matrix to obtain the motion template matrix, which is then input into the cross-attention layer for computation. The output of the cross-attention layer is then input into the temporal attention layer for fusion operations, such as addition. The purpose of adding the motion template matrix to the UNET network's temporal attention layer via the cross-attention layer is to convert the scale of the motion template matrix so that it matches the temporal attention layer, thereby better integrating the information of the motion template matrix into the UNET network's temporal attention layer.

[0065] In one embodiment, the matrix scale of the operation can be expressed as: Select coefficient matrix ∈ R 1×4 , weight matrix ∈ R h×w×d Therefore, in order to perform multiplication operations, the selection coefficient matrix can be expanded in dimension. The expanded scale is expressed as: selection coefficient matrix ∈ R 1×1×4 , then the motion template matrix ∈R output after multiplication h×w×d , the output of the cross attention layer ∈ R h′×w′×d′ It should be noted that dimensionality expansion is only for calculation and does not modify the amount of information represented by the matrix.

[0066] The motion trends of medical animations generated by existing technologies often do not conform to the natural movements of blood vessels and internal organs in reality, resulting in a lack of realism and credibility. It is also difficult to accurately simulate the pulsation of medical parts such as blood vessels and internal organs, which creates obstacles to conveying accurate medical knowledge and understanding the physiological mechanisms of the human body.

[0067] The technical solution of the embodiment of the present application is to classify the static images in the training data to determine the selection coefficient matrix representing the types of different medical parts in the static images, then fuse the selection coefficient matrix with the pre-initialized weight matrix to obtain a motion template matrix, and then add the motion template matrix to the temporal attention layer of the UNET network. During the training process, the parameters of the spatial attention layer of the UNET network are frozen, and the parameters of the temporal attention layer and weight matrix of the UNET network are optimized. Thus, the weight matrix is ​​used to learn the motion trend information representing different medical parts, so that the trained video generation model can generate more realistic animated videos of medical parts based on the weight matrix, and can accurately capture and reproduce the natural motion trends of medical parts such as blood vessels and internal organs, ensure that the animation content is consistent with medical common sense and human physiological principles, and enhance the authenticity and educational value of the animation.

[0068] Figure 3Schematic diagram of another training method for a video generation model provided in an embodiment of the present application. As shown in the figure, E represents a VAE encoder, D represents a VAE decoder, TL represents a temporal attention layer, SL represents a spatial attention layer, and C represents a splicing operation. Wherein, TL and SL form a UNET network. The input static image is first subjected to semantic segmentation and image classification, and then to a motion template selector, and finally added to each TL. At the same time, the input static image is first encoded by E, and noise is added, the image potential representation and the noise potential representation are spliced, and the obtained result is input into the UNET network, and finally the output of the UNET network is input into D to generate the current video. Thus, by means of supervised training, the parameters of SL are frozen during the training process, and the parameters of TL and the weight matrix are continuously optimized. By optimizing the weight matrix parameters, it is possible to learn the real motion trend in the medical part video and save the information representing the motion trend. Wherein, the role played by the motion template selector is to multiply the selection coefficient matrix obtained by image classification with the weight matrix, and then add it to the temporal attention layer of the UNET network through a cross attention layer (not shown in the figure).

[0069] Figure 4 This is a flow chart of a video generation method provided in an embodiment of the present application. This embodiment is applicable to the case where a trained video generation model is used to generate an animated video of a medical part. The method can be executed by a video generation device, which can be implemented in software and / or hardware, and is preferably configured in an electronic device, such as a computer device or a server. Figure 4 As shown, the method includes:

[0070] S401: Obtain a video generation model and a trained weight matrix.

[0071] Among them, the video generation model is trained according to the training method of the video generation model described in any of the embodiments of the present application, and the video generation model includes a VAE encoder, a VAE decoder and a UNET network.

[0072] S402 : performing image classification on a target image containing different medical parts to obtain a target selection coefficient matrix representing the types of different medical parts in the target image.

[0073] S403 , fusing the target selection coefficient matrix with the trained weight matrix to obtain a target motion template matrix, wherein the weight matrix is ​​used to represent motion trend information of different medical parts.

[0074] For example, the target selection coefficient matrix is ​​multiplied by the trained weight matrix to obtain the target motion template matrix.

[0075] S404: Input the target image into the VAE encoder, and output the target encoded latent representation through the VAE encoder.

[0076] S405. Input the target encoding potential representation into the UNET network, and add the target motion template matrix to the temporal attention layer of the UNET network.

[0077] S406: Input the output result of the UNET network with the motion template matrix into the VAE decoder, and generate the target video through the VAE decoder.

[0078] Specifically, after training, the weight matrix learns the real motion trend in the medical part video and saves the information representing the motion trend. Therefore, in the inference stage, the target selection coefficient matrix is ​​directly multiplied by the trained weight matrix to obtain the target motion template matrix, and then the target motion template matrix is ​​added to the temporal attention layer of the UNET network for fusion operation, so that the feature map output by the UNET network saves the information of the motion trend of the medical part. Then, the video is generated by the VAE decoder, and the animation in the video is consistent with the real natural motion trend of the medical part, thereby improving the accuracy and scientificity of the generated animation of the medical part.

[0079] Figure 5 A schematic diagram of an animation production process provided for an embodiment of the present application. As shown in the figure, in this process, the input original image is first semantically segmented to obtain medical part images, and then the medical part images are classified, and different motion templates are applied according to the classification results. For example, for "tubular" medical parts, a peristaltic effect motion template is applied; for "visceral" medical parts, a pulsating effect motion template is applied; for "cell" medical parts, a fluttering effect motion template is applied; for "other" medical parts, no motion template is applied, and they are regarded as stationary. The peristaltic template, the pulsating template, and the fluttering template are the weight matrices described in the embodiment of the present application. The weight matrix is ​​a vector matrix. These three vector matrices are trained in the training phase and frozen in the inference phase. The frozen vector matrix preserves the true motion trend information of different medical parts. Then, a template is selected according to the classification result of the image, that is, the coefficient matrix is ​​multiplied by the weight matrix to obtain a motion template matrix. The input original image passes through the traditional VAE encoder to obtain the image latent representation and noise latent representation, and passes through a UNET network consisting of traditional temporal attention layer and spatial attention layer. At the same time, the motion template matrix is ​​added to the temporal attention layer by the cross attention layer. The feature map output by UNET passes through the traditional VAE decoder to obtain the final video frame.

[0080] Figure 6A schematic diagram of a training device for a video generation model provided in an embodiment of the present application. The video generation model includes a pre-trained VAE encoder, a VAE decoder, and a UNET network, wherein the UNET network includes multiple attention layers consisting of a temporal attention layer and a spatial attention layer. Figure 6 As shown, the training device 60 of the video generation model includes:

[0081] An image classification module 610 is configured to determine, for training data consisting of static images containing different medical parts, a selection coefficient matrix representing the types of the different medical parts in the static images by image classification;

[0082] A motion template selection module 620 is configured to fuse the selection coefficient matrix with a pre-initialized weight matrix to obtain a motion template matrix, wherein the weight matrix is ​​used to represent motion trend information of the different medical parts;

[0083] A first input module 630 is configured to input the static image into the VAE encoder and output an encoded latent representation through the VAE encoder;

[0084] A motion template adding module 640 is configured to input the encoded latent representation into the UNET network and add the motion template matrix into the temporal attention layer of the UNET network;

[0085] A second input module 650 is configured to input the output result of the UNET network with the motion template matrix added thereto into the VAE decoder, and generate the current video through the VAE decoder;

[0086] The optimization training module 660 is used to calculate the current loss based on the real video corresponding to the static image and the current video, and optimize the parameters of the temporal attention layer and the weight matrix of the UNET network according to the current loss, wherein the parameters of the spatial attention layer of the UNET network are frozen during the training process.

[0087] In some embodiments, the image classification module 610 includes:

[0088] a semantic segmentation unit, configured to perform semantic segmentation on the static image to obtain a plurality of medical part images;

[0089] The image classification unit is configured to perform image classification on the plurality of medical part images using an image classification model, and determine a selection coefficient matrix representing the types to which the plurality of medical parts belong.

[0090] In some embodiments, the motion template selection module 620 is specifically configured to:

[0091] The selection coefficient matrix is ​​multiplied by a pre-initialized weight matrix to obtain a motion template matrix.

[0092] In some embodiments, the motion template adding module 640 is specifically configured to:

[0093] The motion template matrix is ​​added to the temporal attention layer of the UNET network through the cross attention layer.

[0094] In some embodiments, the first input module 630 includes:

[0095] An input unit, configured to input the static image into the VAE encoder to obtain a potential representation of the image;

[0096] a noise adding unit, configured to add noise to the feature map of the static image obtained by the VAE encoder to obtain a noise potential representation;

[0097] A splicing unit is configured to splice the image potential representation and the noise potential representation to obtain the coding potential representation.

[0098] The training device for the video generation model provided in the embodiment of the present application can be used to execute the technical solution of the training method for the video generation model in the above embodiment. Its implementation principle and technical effects are similar and will not be repeated here.

[0099] Figure 7 This is a schematic diagram of the structure of a video generation device provided in an embodiment of the present application. Figure 7 As shown, the video generating device 70 includes:

[0100] A model acquisition module 710 is configured to acquire a video generation model and a trained weight matrix, wherein the video generation model is trained by the video generation model training apparatus according to claim 8, and the video generation model includes a VAE encoder, a VAE decoder, and a UNET network;

[0101] The target selection coefficient matrix determination module 720 is configured to classify the target image containing different medical parts to obtain a target selection coefficient matrix representing the types of the different medical parts in the target image;

[0102] A target motion template matrix determination module 730 is configured to fuse the target selection coefficient matrix with the trained weight matrix to obtain a target motion template matrix, wherein the weight matrix is ​​used to represent motion trend information of different medical parts;

[0103] a target encoding latent representation determination module 740, configured to input the target image into the VAE encoder and output a target encoding latent representation via the VAE encoder;

[0104] A target motion template adding module 750 is used to input the target encoding potential representation into the UNET network and add the target motion template matrix to the temporal attention layer of the UNET network;

[0105] The target video generation module 760 is used to input the output result of the UNET network with the motion template matrix added into the VAE decoder, and generate the target video through the VAE decoder.

[0106] In some embodiments, the target motion template matrix determination module 730 is specifically configured to:

[0107] The target selection coefficient matrix is ​​multiplied by the trained weight matrix to obtain the target motion template matrix.

[0108] The video generation device provided in the embodiment of the present application can be used to execute the technical solution of the video generation method in the above embodiment. Its implementation principle and technical effects are similar and will not be repeated here.

[0109] It should be noted that the division of the various modules of the above device is merely a division of logical functions. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can all be implemented in the form of software called by a processing element; or all be implemented in the form of hardware; or some modules can be implemented in the form of software called by a processing element, and some modules can be implemented in the form of hardware. For example, the motion template selection module 620 can be a separate processing element, or it can be integrated into a chip of the above device. In addition, it can be stored in the form of program code in the memory of the above device, and called by a processing element of the above device to perform the functions of the above motion template selection module 620. The implementation of other modules is similar. In addition, these modules can all or partly be integrated together, or they can be implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. During implementation, each step of the above method or each of the above modules can be completed by hardware integrated logic circuits in the processor element or by software instructions.

[0110] Figure 8 This is a schematic diagram of the structure of the electronic device provided in the embodiment of the present application. Figure 8 As shown, the electronic device may include: a transceiver 121 , a processor 122 , and a memory 123 .

[0111] The processor 122 executes the computer-executable instructions stored in the memory, so that the processor 122 implements the solutions in the above embodiments. The processor 122 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0112] The memory 123 is connected to the processor 122 via a system bus and communicates with the processor 122. The memory 123 is used to store computer program instructions.

[0113] The transceiver 121 may be used to obtain tasks to be executed and configuration information of the tasks to be executed.

[0114] The system bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, among others. The system bus can be divided into an address bus, a data bus, a control bus, and so on. For ease of illustration, the figure shows only one thick line, but this does not imply that there is only one bus or only one type of bus. Transceivers are used to enable communication between the database access device and other computers (e.g., clients, read-write libraries, and read-only libraries). Memory may include random access memory (RAM) and non-volatile memory.

[0115] The electronic device provided in the embodiment of the present application may be the terminal device of the above embodiment.

[0116] An embodiment of the present application also provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed on a computer, the computer executes the technical solutions of the video generation model training method and the video generation method described in the above embodiments.

[0117] An embodiment of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When the at least one processor executes the computer program, the technical solutions of the video generation model training method and the video generation method described in the above embodiments can be implemented.

[0118] The computer program product, during implementation, may be written in one or more programming languages ​​or a combination thereof, for performing the operations of the present application, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0119] Note that the above are only preferred embodiments of the present application and the technical principles employed. Those skilled in the art will understand that the present application is not limited to the specific embodiments herein, and that various obvious changes, readjustments, and substitutions can be made by those skilled in the art without departing from the scope of protection of the present application. Therefore, although the present application has been described in more detail through the above embodiments, the present application is not limited to the above embodiments and may include many other equivalent embodiments without departing from the scope of the present application. The scope of the present application is determined by the scope of the appended claims.

Claims

1. A training method for a video generation model, characterized in that: The video generation model includes a pre-trained VAE encoder, a VAE decoder, and a UNET network, wherein the UNET network includes multiple attention layers consisting of temporal attention layers and spatial attention layers, and the method includes: For training data consisting of static images containing different medical parts, determining a selection coefficient matrix representing the types of different medical parts in the static images by image classification; fusing the selection coefficient matrix with a pre-initialized weight matrix to obtain a motion template matrix, wherein the weight matrix is ​​used to characterize the motion trend information of the different medical parts; Input the static image into the VAE encoder, and output the encoded latent representation through the VAE encoder; Inputting the encoded latent representation into the UNET network and adding the motion template matrix into the temporal attention layer of the UNET network; Inputting the output result of the UNET network to which the motion template matrix is ​​added into the VAE decoder, and generating the current video through the VAE decoder; A current loss is calculated based on a real video corresponding to the static image and the current video, and parameters of the temporal attention layer and the weight matrix of the UNET network are optimized according to the current loss, wherein the parameters of the spatial attention layer of the UNET network are frozen during training.

2. The method according to claim 1, characterized in that The determining of the selection coefficient matrix representing the types of different medical parts in the static image by image classification includes: Performing semantic segmentation on the static image to obtain multiple medical part images; An image classification model is used to perform image classification on the multiple medical part pictures, and a selection coefficient matrix representing the types to which the multiple medical parts belong is determined.

3. The method according to claim 1, characterized in that The step of fusing the selection coefficient matrix with a pre-initialized weight matrix to obtain a motion template matrix includes: The selection coefficient matrix is ​​multiplied by a pre-initialized weight matrix to obtain a motion template matrix.

4. The method according to claim 1, wherein Adding the motion template matrix to the temporal attention layer of the UNET network includes: The motion template matrix is ​​added to the temporal attention layer of the UNET network through the cross attention layer.

5. The method according to claim 1, wherein Inputting the static image into the VAE encoder and outputting the encoded latent representation through the VAE encoder includes: Inputting the static image into the VAE encoder to obtain a potential representation of the image; Adding noise to the feature map of the static image obtained by the VAE encoder to obtain a noise latent representation; The image latent representation and the noise latent representation are concatenated to obtain the encoded latent representation.

6. A video generation method, characterized in that: include: Obtaining a video generation model and a trained weight matrix, wherein the video generation model is trained according to the video generation model training method according to any one of claims 1 to 5, and the video generation model includes a VAE encoder, a VAE decoder, and a UNET network; performing image classification on a target image containing different medical parts to obtain a target selection coefficient matrix representing the types of the different medical parts in the target image; fusing the target selection coefficient matrix with the trained weight matrix to obtain a target motion template matrix, wherein the weight matrix is ​​used to characterize the motion trend information of the different medical parts; Input the target image into the VAE encoder, and output the target encoded latent representation through the VAE encoder; Inputting the target encoding potential representation into the UNET network, and adding the target motion template matrix into the temporal attention layer of the UNET network; The output result of the UNET network to which the motion template matrix is ​​added is input into the VAE decoder, and the target video is generated by the VAE decoder.

7. The method according to claim 6, characterized in that The target selection coefficient matrix is ​​fused with the trained weight matrix to obtain a target motion template matrix, including: The target selection coefficient matrix is ​​multiplied by the trained weight matrix to obtain the target motion template matrix.

8. A training device for a video generation model, characterized in that: The video generation model includes a pre-trained VAE encoder, a VAE decoder and a UNET network, wherein the UNET network includes multiple attention layers consisting of a temporal attention layer and a spatial attention layer, and the apparatus includes: An image classification module is configured to determine, for training data consisting of static images containing different medical parts, a selection coefficient matrix representing the types of the different medical parts in the static images by image classification; a motion template selection module, configured to fuse the selection coefficient matrix with a pre-initialized weight matrix to obtain a motion template matrix, wherein the weight matrix is ​​used to characterize motion trend information of the different medical parts; A first input module is configured to input the static image into the VAE encoder and output an encoded latent representation through the VAE encoder; A motion template adding module, configured to input the encoded potential representation into the UNET network and add the motion template matrix into the temporal attention layer of the UNET network; A second input module is used to input the output result of the UNET network added with the motion template matrix into the VAE decoder, and generate the current video through the VAE decoder; An optimization training module is used to calculate a current loss based on a real video corresponding to the static image and the current video, and optimize the parameters of the temporal attention layer and the weight matrix of the UNET network according to the current loss, wherein the parameters of the spatial attention layer of the UNET network are frozen during the training process.

9. A video generating device, characterized in that: include: A model acquisition module, configured to acquire a video generation model and a trained weight matrix, wherein the video generation model is trained by the video generation model training device according to claim 8, and the video generation model includes a VAE encoder, a VAE decoder, and a UNET network; a target selection coefficient matrix determination module, configured to classify target images containing different medical parts to obtain a target selection coefficient matrix representing the types of the different medical parts in the target image; a target motion template matrix determination module, configured to fuse the target selection coefficient matrix with the trained weight matrix to obtain a target motion template matrix, wherein the weight matrix is ​​used to characterize motion trend information of the different medical parts; a target encoding potential representation determination module, configured to input the target image into the VAE encoder and output the target encoding potential representation through the VAE encoder; A target motion template adding module is used to input the target encoding potential representation into the UNET network and add the target motion template matrix to the temporal attention layer of the UNET network; The target video generation module is used to input the output result of the UNET network with the motion template matrix added into the VAE decoder, and generate the target video through the VAE decoder.

10. An electronic device, characterized in that: include: a processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the training method of the video generation model as described in any one of claims 1 to 5, and to implement the video generation method as described in claim 6 or 7.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the training method for the video generation model as described in any one of claims 1 to 5, and to implement the video generation method as described in claim 6 or 7.

12. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, implements the training method of the video generation model as described in any one of claims 1 to 5, and implements the video generation method as described in claim 6 or 7.

Images