A video generation method, device, apparatus and storage medium thereof

Abstract

The embodiment of the application belongs to the technical field of image processing, is applied to a video generation scene with images, and relates to a video generation method, device and equipment and a storage medium thereof, wherein a material set for video generation is acquired; material image, text embedding data and description voice data are mapped and sorted among images, texts and voices; text embedding data respectively needed to be added by each material image in a video generation process and a description voice duration respectively corresponding to each material image are identified; the description voice duration is taken as a video generation guidance parameter and added to a preset video generation model; and the material set is input into the video generation model to generate a target video in combination with the video generation guidance parameter. The video generation method is applied to a medical experiment or a teaching video generation scene, can assist medical education and scientific research, and improves a guidance video generation mode in medical education and scientific research.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and is applied to scenarios where images are used to generate videos. In particular, it relates to a video generation method, apparatus, device, and storage medium thereof. Background Technology

[0002] With the rapid development of deep learning technology, diffusion models have made significant progress in the field of video generation. These models generate video content frame by frame by frame, creating more coherent and natural video content. By introducing long-term dependencies, diffusion models can capture more complex spatiotemporal relationships in videos, making the generated videos more narrative and emotionally resonant.

[0003] In the medical field, video generation technology holds significant importance. For example, the medical field requires substantial training and educational resources, and video generation technology can be used to create realistic medical case demonstration videos, providing medical students and practitioners with more vivid and intuitive learning resources. How to generate instructional videos for medical education and research has become a pressing issue that needs to be addressed. Summary of the Invention

[0004] The purpose of this application is to provide a video generation method, apparatus, device and storage medium to solve the problem of how to generate instructional videos for medical education and research.

[0005] To address the aforementioned technical problems, this application provides a video generation method, which employs the following technical solution:

[0006] A video generation method includes the following steps:

[0007] Obtain a set of materials for video generation, wherein the set of materials includes material images, text embedding data corresponding to each material image, and descriptive speech data;

[0008] All material images in the material set are sorted according to a preset sorting rule, and a sequence number is set for all material images based on the sorting results.

[0009] Based on the sequence number, the image material, the embedded text data, and the descriptive speech data are mapped and organized to obtain the mapping and organization result;

[0010] Based on the mapping and processing results, the text embedding data that needs to be added to each material image during the video generation process, as well as the duration of the descriptive speech corresponding to each material image, are identified.

[0011] The duration of the descriptive audio corresponding to each source image is used as a video generation guide parameter and added to the preset video generation model;

[0012] The source material set is input into the video generation model, and the target video is generated by combining the video generation guidance parameters.

[0013] Furthermore, the step of obtaining the material set for video generation specifically includes:

[0014] The photographic images obtained from medical scientific experiments and used for medical guidance and teaching are used as the source images. The shooting mode selected during the shooting includes adding shooting time information to each photographic image.

[0015] The introductory text content added for each photographic image is obtained as the text embedding data;

[0016] The introductory language content recorded for each photographic image is obtained as the descriptive speech data.

[0017] Furthermore, the step of sorting all material images in the material set according to a preset sorting rule, and setting sequence numbers for all material images based on the sorting results, specifically includes:

[0018] The media images in the media set are sorted based on the shooting time information corresponding to each photographic image. The preset sorting rules include sorting the photographic images according to the order of shooting time information.

[0019] Based on the sorting results, the position information of each of the source images in the sorting sequence is identified, and the position information is used as the sequence number.

[0020] Furthermore, the step of identifying the text embedding data that needs to be added to each source image during video generation, and the corresponding descriptive speech duration for each source image, based on the mapping and processing results, specifically includes:

[0021] Based on the mapping and processing results, the text embedding data corresponding to each material image is identified;

[0022] The text embedding data is set as the text embedding data that needs to be added to the corresponding source image during the video generation process;

[0023] Based on the mapping and processing results, the descriptive speech data corresponding to each source image is identified;

[0024] Set the duration of the descriptive voice data to the duration of the descriptive voice for the corresponding source image.

[0025] Furthermore, the step of adding the duration of the descriptive audio corresponding to each source image as a video generation guidance parameter to the preset video generation model specifically includes:

[0026] Based on the duration of the descriptive audio corresponding to each image, the video duration corresponding to each image is determined after image diffusion processing using the image diffusion generation method.

[0027] The total audio duration is obtained by summing the description audio durations corresponding to all the images in the material set, and the total audio duration is used as the total video duration of the target video generated after processing by the video generation model.

[0028] The video duration corresponding to each source image and the total video duration are used as video generation guidance parameters and deployed into the video generation model.

[0029] Furthermore, the video generation model includes an image diffusion-based video generation model, which includes a transformer encoding / decoding structure and a U-Net intermediate layer structure. The step of inputting the source set into the video generation model and generating the target video in conjunction with the video generation guidance parameters specifically includes:

[0030] The encoding layer in the transformer codec structure encodes the material images, text embedding data, and descriptive speech data in the material set, respectively, to obtain the corresponding image encoding features, text encoding features, and speech encoding features.

[0031] According to the sequence number, the previous condition image and the next condition image are obtained from the source images in sequence to form N-1 groups of images. The sequence number of the previous condition image is 1 less than the sequence number of the next condition image. N is a positive integer, representing the total number of source images in the source set.

[0032] Based on the different groups of images, N-1 groups of image coding features are obtained. Each group of images includes two source images with a sequence number difference of 1. Each group of image coding features includes the image coding features corresponding to the two source images respectively.

[0033] By combining a preset time step, the N-1 groups of image coding features are embedded with noise features in an iterative manner to obtain the image coding features with embedded noise features corresponding to the N-1 groups of image coding features respectively.

[0034] The image coding features embedded with noise features are processed by an attention mechanism and then mapped to the intermediate layer structure of the U-Net for image coding feature denoising to obtain the denoised image coding features.

[0035] The denoised image encoding features, text encoding features, and speech encoding features are input into the decoding layer of the transformer encoding / decoding structure. Combined with the video generation guidance parameters and the image-text-speech mapping relationship, the target video is generated.

[0036] Furthermore, the step of generating the target video by combining the video generation guidance parameters and the image-text-speech mapping relationship specifically includes:

[0037] The display duration of each video image in the target video is controlled according to the video generation guidance parameters;

[0038] Based on the image-text-speech mapping relationship, the text data output from the decoding layer is added to the reserved text addition position in the corresponding image; and,

[0039] Based on the image-text-speech mapping relationship, the speech data output by the decoding layer is added to the display time period corresponding to the image.

[0040] To address the aforementioned technical problems, this application also provides a video generation apparatus, which employs the following technical solution:

[0041] A video generation apparatus, comprising:

[0042] The material set acquisition module is used to acquire a material set for video generation, wherein the material set includes material images, text embedding data corresponding to each material image, and descriptive speech data;

[0043] The image serialization processing module is used to sort all material images in the material set according to a preset sorting rule, and set a sequence number for all material images based on the sorting results;

[0044] The mapping and processing module is used to perform image-text-speech mapping and processing on the material image, the text embedding data, and the descriptive speech data based on the sequence number, and obtain the mapping and processing result;

[0045] The mapping relationship recognition module is used to identify, based on the mapping and sorting results, the text embedding data that needs to be added to each material image during the video generation process, and the duration of the descriptive speech corresponding to each material image;

[0046] The video generation model parameter adding module is used to add the duration of the descriptive audio corresponding to each source image as a video generation guidance parameter to the preset video generation model;

[0047] The target video generation module is used to input the material set into the video generation model and generate the target video in combination with the video generation guidance parameters.

[0048] To address the aforementioned technical problems, this application also provides a computer device that employs the following technical solution:

[0049] A computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the video generation method described above.

[0050] To address the aforementioned technical problems, this application also provides a computer-readable storage medium, employing the technical solution described below:

[0051] A computer-readable storage medium storing computer-readable instructions that, when executed by a processor, implement the steps of the video generation method described above.

[0052] Compared with the prior art, the embodiments of this application have the following main advantages:

[0053] The video generation method described in this application involves: acquiring a set of source materials for video generation; sorting all source images in the source material set and assigning sequence numbers; mapping the source images, text embedding data, and descriptive speech data based on the sequence numbers; identifying the text embedding data to be added to each source image during video generation, as well as the corresponding descriptive speech duration for each source image, based on the mapping results; adding the corresponding descriptive speech duration for each source image as a video generation guidance parameter to a preset video generation model; inputting the source material set into the video generation model and generating the target video in conjunction with the video generation guidance parameters. Applying this video generation method to medical experimental or educational video generation scenarios can assist medical education and research, improving the way guidance videos are generated in medical education and research. Attached Figure Description

[0054] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 This is an exemplary system architecture diagram to which this application can be applied;

[0056] Figure 2 This is a flowchart of an embodiment of the video generation method according to this application;

[0057] Figure 3 yes Figure 2 A flowchart of a specific embodiment of step 201 shown;

[0058] Figure 4 yes Figure 2 A flowchart of a specific embodiment of step 202 shown;

[0059] Figure 5 yes Figure 2 A flowchart of a specific embodiment of step 204 shown;

[0060] Figure 6 yes Figure 2 A flowchart of a specific embodiment of step 205 shown;

[0061] Figure 7 yes Figure 2 A flowchart of a specific embodiment of step 206 shown;

[0062] Figure 8 yes Figure 7 A flowchart of a specific embodiment of step 706 shown;

[0063] Figure 9 This is a schematic diagram of the structure of an embodiment of the video generation apparatus according to this application;

[0064] Figure 10 This is a schematic diagram of the structure of one embodiment of the computer device according to this application. Detailed Implementation

[0065] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0066] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0067] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0068] like Figure 1 As shown, system architecture 100 may include terminal device 101, network 102, and server 103. Terminal device 101 may be a laptop 1011, tablet 1012, or mobile phone 1013. Network 102 is used as a medium to provide a communication link between terminal device 101 and server 103. Network 102 may include various connection types, such as wired, wireless communication links, or fiber optic cables.

[0069] Users can use terminal device 101 to interact with server 103 via network 102 to receive or send messages, etc. Various communication client applications can be installed on terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0070] Terminal device 101 can be various electronic devices with a display screen and support web browsing. In addition to laptops 1011, tablets 1012, or mobile phones 1013, terminal device 101 can also be an e-book reader, MP3 player (Moving Picture Experts Group Audio Layer III), MP4 player (Moving Picture Experts Group Audio Layer IV), laptop computer, and desktop computer, etc.

[0071] Server 103 can be a server that provides various services, such as a backend server that provides support for the pages displayed on terminal device 101.

[0072] It should be noted that the video generation method provided in this application embodiment is generally executed by a terminal device, and correspondingly, the video generation device is generally located in the terminal device.

[0073] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0074] Continue to refer to Figure 2 A flowchart of an embodiment of the video generation method according to this application is shown. The video generation method includes the following steps:

[0075] Step 201: Obtain a set of materials for video generation, wherein the set of materials includes material images, text embedding data corresponding to each material image, and descriptive speech data;

[0076] Specifically, the source images include medical education images and / or medical research images, such as experimental operation images obtained according to each step of the PCR experiment. The embedded text data includes key points of the experimental operation, reagent addition amount, operation precautions and reaction mechanism descriptions corresponding to each experimental operation image. The descriptive voice data includes simple audio introductions of the experimental steps.

[0077] By acquiring a set of materials for video generation, subsequent video generation can be facilitated. Applying the video generation method described in this application to medical experiment or teaching video generation scenarios can assist medical education and research, and improve the way guidance videos are generated in medical education and research.

[0078] Step 202: Sort all material images in the material set according to the preset sorting rules, and set sequence numbers for all material images based on the sorting results;

[0079] By sorting all the material images in the material set and assigning sequence numbers to all the material images based on the sorting results, the final target video is generated according to specific steps. Assuming that the material images are experimental operation images obtained according to each operation step in a PCR experiment, the purpose of sorting is to ensure the correctness of the steps in the generated medical experiment or teaching video.

[0080] Step 203: Based on the sequence number, perform image-text-speech mapping on the source image, the embedded text data, and the descriptive speech data to obtain the mapping result;

[0081] The sequence number is used to map and organize the source images, embedded text data, and descriptive speech data into images, text, and speech, combining text and speech descriptions to generate a target video, which can then be used for teaching demonstrations.

[0082] Step 204: Based on the mapping and processing results, identify the text embedding data that needs to be added to each material image during the video generation process, as well as the duration of the descriptive speech corresponding to each material image;

[0083] Specifically, by identifying the duration of the accompanying audio description for each image, the aim is to control the display duration of each image in the final target video, thus avoiding a mismatch between image display and audio description.

[0084] Step 205: The duration of the descriptive audio corresponding to each source image is used as a video generation guidance parameter and added to the preset video generation model;

[0085] By adding the duration of the descriptive audio corresponding to each source image as a guiding parameter for video generation to the preset video generation model, the aim is to control the display duration of each source image in the final target video using a model-based generation method, thereby avoiding a mismatch between image display and audio description.

[0086] Step 206: Input the material set into the video generation model and generate the target video by combining the video generation guidance parameters.

[0087] In this embodiment, a material set for video generation is obtained; all material images in the material set are sorted and assigned sequence numbers; based on the sequence numbers, the material images, text embedding data, and descriptive speech data are mapped and organized into image-text-speech pairs; according to the mapping and organization results, the text embedding data that needs to be added to each material image during video generation, as well as the corresponding descriptive speech duration for each material image, are identified; the descriptive speech duration corresponding to each material image is used as a video generation guidance parameter and added to a preset video generation model; the material set is input into the video generation model, and the target video is generated by combining the video generation guidance parameters. Applying the video generation method described in this application to medical experiment or teaching video generation scenarios can assist medical education and research, improving the way guidance videos are generated in medical education and research.

[0088] Continue to refer to Figure 3 In some optional implementations, Figure 3 yes Figure 2 A flowchart of a specific embodiment of step 201 shown includes the following steps:

[0089] Step 301: Obtain photographic images for medical guidance and teaching from the medical science experiment process through shooting, wherein the shooting mode selected during shooting includes adding shooting time information to each photographic image;

[0090] By setting the shooting mode selected during shooting, you can add shooting time information to each photograph to facilitate subsequent sorting and processing of each source image.

[0091] Step 302: Obtain the introductory text content added for each photographic image as the text embedding data;

[0092] By obtaining the introductory text content added to each photographic image as the text embedding data, it is possible to add textual descriptions to the corresponding images in the video during video generation.

[0093] Step 303: Obtain the introductory language content recorded for each photographic image as the descriptive speech data.

[0094] By obtaining introductory language content recorded separately for each photographic image, it is possible to add descriptive voice to each image in the subsequently generated video, and also to control the display duration of each image in the video based on the descriptive voice data.

[0095] Continue to refer to Figure 4 In some optional implementations, Figure 4 yes Figure 2 A flowchart of a specific embodiment of step 202 shown includes the following steps:

[0096] Step 401: Sort all material images in the material set based on the shooting time information corresponding to each photographic image, wherein the preset sorting rule includes sorting the photographic images according to the order of shooting time information.

[0097] Step 402: Based on the sorting process results, identify the position information of each of the source images in the sorting sequence, and use the position information as the sequence number.

[0098] By sorting all the images in the material set and assigning sequence numbers to all the images based on the sorting results, the final generated target video follows specific steps, ensuring the correctness of the steps in the generated medical experiment or teaching video.

[0099] Continue to refer to Figure 5 In some optional implementations, Figure 5 yes Figure 2 A flowchart of a specific embodiment of step 204 shown includes the following steps:

[0100] Step 501: Based on the mapping and sorting results, identify the text embedding data corresponding to each material image;

[0101] Step 502: Set the text embedding data as the text embedding data that needs to be added to the corresponding material image during the video generation process;

[0102] Step 503: Based on the mapping and processing results, identify the descriptive speech data corresponding to each source image;

[0103] Step 504: Set the duration of the descriptive speech data to the duration of the descriptive speech for the corresponding source image.

[0104] By identifying the text embedding data that needs to be added to each source image during video generation, as well as the corresponding description speech duration for each source image, the text embedding data can be added to the video during subsequent target video generation, and the total length of the target video and the display length of each image can be controlled by combining the description speech duration.

[0105] Continue to refer to Figure 6 In some optional implementations, Figure 6 yes Figure 2 A flowchart of a specific embodiment of step 205 shown includes the following steps:

[0106] Step 601: Based on the duration of the descriptive speech corresponding to each material image, determine the video duration corresponding to each material image after image diffusion processing using the image diffusion generation method;

[0107] Step 602: Accumulate the duration of the descriptive speech corresponding to all material images in the material set to obtain the total duration of the speech, and use the total duration of the speech as the total duration of the target video generated after processing by the video generation model;

[0108] Step 603: The video duration corresponding to each source image and the total video duration are used as video generation guidance parameters and deployed into the video generation model.

[0109] By adding the duration of the descriptive speech corresponding to each source image as a guiding parameter for video generation to the preset video generation model, the generation of the video can be controlled according to the duration of the descriptive speech, preventing the target video from being too short or too long, and ensuring the appropriate length of the generated target video.

[0110] In this embodiment, the video generation model includes an image diffusion-based video generation model, which includes a transformer encoding / decoding structure and a U-Net intermediate layer structure. Specifically, the encoding layer in the transformer encoding / decoding structure can be directly used as the input layer of the video generation model, and the decoding layer in the transformer encoding / decoding structure can be used as the output layer of the video generation model. The U-Net intermediate layer structure is used to perform an image diffusion generation step on the source image and is set between the encoding layer and the decoding layer.

[0111] Continue to refer to Figure 7 In some optional implementations, Figure 7 yes Figure 2 A flowchart of a specific embodiment of step 206 shown includes the following steps:

[0112] Step 701: Through the encoding layer in the transformer codec structure, the material images, text embedding data and descriptive speech data in the material set are encoded to obtain the corresponding image encoding features, text encoding features and speech encoding features.

[0113] The corresponding image coding features, text coding features, and speech coding features are obtained through the coding layer, so as to facilitate subsequent decoding and integration of the image coding features, text coding features, and speech coding features.

[0114] Step 702: According to the sequence number, the previous condition image and the next condition image are obtained from the material images in sequence to form N-1 groups of images. The sequence number of the previous condition image is 1 less than the sequence number of the next condition image. N is a positive integer, representing the total number of material images in the material set.

[0115] Specifically, for example: based on the experimental operation images obtained from each step of the PCR experiment, a total of N=15 images were taken from the start to the end of the experiment, with sequence numbers from 1 to 15. First, the image with sequence number 1 is used as the preceding condition image, and the image with sequence number 2 is used as the following condition image to generate the first set of images. Then, the image with sequence number 2 is used as the preceding condition image, and the image with sequence number 3 is used as the following condition image to generate the second set of images, and so on, until the last set of images is obtained with the image with sequence number 14 as the preceding condition image and the image with sequence number 15 as the following condition image. Thus, a total of N-1, or 14 sets of images are obtained.

[0116] By constructing group images, the changes from one image to the next are introduced as much as possible during target video generation based on the group images and image diffusion generation method, so that the generated target video pays more attention to the changes between images.

[0117] Step 703: Based on the different groups of images, obtain N-1 groups of image coding features, wherein a group of images includes two source images with a sequence number difference of 1, and each group of image coding features includes the image coding features corresponding to the two source images respectively;

[0118] Step 704: Combined with a preset time step, the N-1 groups of image coding features are embedded with noise features in an iterative manner to obtain the image coding features with embedded noise features corresponding to the N-1 groups of image coding features respectively;

[0119] Specifically, since the number of source images for the generated video is fixed, an image diffusion generation method is adopted. By introducing noise features in an iterative manner according to a preset time step, image coding features embedded with noise features are obtained. Assuming that for the source image with sequence number 2, the time step is 0.2 seconds and the total number of iterations is 60, a total of 60 image coding features embedded with noise features are obtained, and each iteration introduces noise features again on the basis of the previous iteration.

[0120] Here, the product of the time step and the number of iterations must not exceed the duration of the descriptive audio corresponding to the source image. That is, assuming the descriptive audio duration corresponding to the source image with sequence number 3 is 20 seconds, and the time step is 0.2 seconds, then the maximum number of iterations is 100. This avoids the amplified image from far exceeding the reserved video display time during image diffusion generation.

[0121] Step 705: The image coding features embedded with noise features are processed by the attention mechanism and mapped to the U-Net intermediate layer structure to perform image coding feature denoising processing to obtain the denoised image coding features;

[0122] Specifically, since the number of source images for the generated video is fixed, an image diffusion generation method is adopted. By introducing noise features in an iterative manner according to a preset time step, and then performing noise reduction processing, it essentially achieves image expansion of the source images, so that the generated target video is generated based on more images.

[0123] Step 706: Input the denoised image coding features, text coding features, and speech coding features into the decoding layer of the transformer codec structure, and generate the target video by combining the video generation guidance parameters and the image-text-speech mapping relationship.

[0124] Continue to refer to Figure 8 In some optional implementations, Figure 8 yes Figure 7 A flowchart of a specific embodiment of step 706 shown includes the following steps:

[0125] Step 801: Control the display duration of each video image in the target video according to the video generation guidance parameters;

[0126] Step 802: Based on the image-text-speech mapping relationship, add the text data output by the decoding layer to the reserved text addition position in the corresponding image; and,

[0127] Step 803: Based on the image-text-speech mapping relationship, add the speech data output by the decoding layer to the display time period corresponding to the image.

[0128] Specifically, the display duration of each video image in the target video is controlled according to the video generation guidance parameters. Based on the image-text-speech mapping relationship, the text data output by the decoding layer is added to the reserved text addition position of the corresponding image. In addition, the speech data output by the decoding layer is added to the display time period corresponding to the image. This not only controls the display duration of the target video generation, but also performs mimicry processing on the static display images, adding text and speech descriptions.

[0129] This application obtains a set of materials for video generation; sorts all images in the set and assigns sequence numbers; performs image-text-speech mapping based on the sequence numbers; identifies the text embedding data and the corresponding speech duration for each image during video generation based on the mapping results; uses the speech duration for each image as a video generation guidance parameter and adds it to a preset video generation model; inputs the set of materials into the video generation model and generates the target video based on the video generation guidance parameters. Applying this video generation method to medical experiment or educational video generation scenarios can assist medical education and research, improving the way guidance videos are generated in medical education and research.

[0130] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0131] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0132] In this embodiment, a material set for video generation is obtained; all material images in the material set are sorted and assigned sequence numbers; based on the sequence numbers, the material images, text embedding data, and descriptive speech data are mapped and organized into image-text-speech pairs; according to the mapping and organization results, the text embedding data that needs to be added to each material image during video generation, as well as the corresponding descriptive speech duration for each material image, are identified; the descriptive speech duration corresponding to each material image is used as a video generation guidance parameter and added to a preset video generation model; the material set is input into the video generation model, and the target video is generated by combining the video generation guidance parameters. Applying the video generation method described in this application to medical experiment or teaching video generation scenarios can assist medical education and research, improving the way guidance videos are generated in medical education and research.

[0133] Further reference Figure 9 As a response to the above Figure 2 To implement the method shown, this application provides an embodiment of a video generation apparatus, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0134] like Figure 9 As shown, the video generation device 900 described in this embodiment includes: a material set acquisition module 901, an image serialization processing module 902, a mapping and organization module 903, a mapping relationship recognition module 904, a video generation model parameter addition module 905, and a target video generation module 906. Wherein:

[0135] The material set acquisition module 901 is used to acquire a material set for video generation, wherein the material set includes material images, text embedding data corresponding to each material image, and descriptive speech data;

[0136] The image serialization processing module 902 is used to sort all material images in the material set according to a preset sorting rule, and set a sequence number for all material images based on the sorting results.

[0137] The mapping and processing module 903 is used to perform image-text-speech mapping and processing on the material image, the text embedding data and the descriptive speech data based on the sequence number, and obtain the mapping and processing result;

[0138] The mapping relationship recognition module 904 is used to identify, based on the mapping and sorting results, the text embedding data that needs to be added to each material image during the video generation process, and the duration of the descriptive speech corresponding to each material image.

[0139] The video generation model parameter adding module 905 is used to add the duration of the descriptive voice corresponding to each source image as a video generation guidance parameter to the preset video generation model;

[0140] The target video generation module 906 is used to input the material set into the video generation model and generate a target video in combination with the video generation guidance parameters.

[0141] This application obtains a set of materials for video generation; sorts all images in the set and assigns sequence numbers; performs image-text-speech mapping based on the sequence numbers; identifies the text embedding data and the corresponding speech duration for each image during video generation based on the mapping results; uses the speech duration for each image as a video generation guidance parameter and adds it to a preset video generation model; inputs the set of materials into the video generation model and generates the target video based on the video generation guidance parameters. Applying this video generation method to medical experiment or educational video generation scenarios can assist medical education and research, improving the way guidance videos are generated in medical education and research.

[0142] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware through computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0143] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0144] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 10 , Figure 10 This is a basic structural block diagram of the computer device in this embodiment.

[0145] The computer device 10 includes a memory 10a, a processor 10b, and a network interface 10c, which are interconnected via a system bus. It should be noted that... Figure 10 Only a computer device 10 with component memory 10a, processor 10b, and network interface 10c is shown. However, it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead. Those skilled in the art will understand that the computer device described herein is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), digital digital processors (DSPs), embedded devices, etc.

[0146] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0147] The memory 10a includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 10a may be an internal storage unit of the computer device 10, such as the hard disk or memory of the computer device 10. In other embodiments, the memory 10a may also be an external storage device of the computer device 10, such as a plug-in hard disk, smart memory card (SMC), secure digital card (SD) card, flash memory card, etc. of the computer device 10. Of course, the memory 10a may also include both the internal storage unit and the external storage device of the computer device 10. In this embodiment, the memory 10a is typically used to store the operating system and various application software installed on the computer device 10, such as computer-readable instructions for a video generation method. Furthermore, the memory 10a can also be used to temporarily store various types of data that have been output or will be output.

[0148] In some embodiments, the processor 10b may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 10b is typically used to control the overall operation of the computer device 10. In this embodiment, the processor 10b is used to execute computer-readable instructions stored in the memory 10a or to process data, for example, to execute computer-readable instructions for the video generation method.

[0149] The network interface 10c may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 10 and other electronic devices.

[0150] The computer device proposed in this embodiment belongs to the field of image processing technology and is applied to video generation scenarios using images. This application obtains a set of materials for video generation; sorts all material images in the set and assigns sequence numbers; performs image-text-speech mapping based on the sequence numbers for the material images, text embedding data, and descriptive speech data; identifies the text embedding data that needs to be added to each material image during video generation, as well as the corresponding descriptive speech duration for each material image, based on the mapping results; adds the corresponding descriptive speech duration for each material image as a video generation guidance parameter to a preset video generation model; inputs the material set into the video generation model and generates the target video by combining the video generation guidance parameters. Applying the video generation method described in this application to medical experiment or teaching video generation scenarios can assist medical education and research, improving the way guidance videos are generated in medical education and research.

[0151] This application also provides another embodiment, namely, providing a computer-readable storage medium storing computer-readable instructions that can be executed by a processor to cause the processor to perform the steps of the video generation method described above.

[0152] The computer-readable storage medium proposed in this embodiment belongs to the field of image processing technology and is applied to video generation scenarios using images. This application obtains a set of materials for video generation; sorts all material images in the set and assigns sequence numbers; performs image-text-speech mapping based on the sequence numbers; identifies the text embedding data to be added to each material image during video generation, as well as the corresponding speech duration for each material image, based on the mapping results; adds the speech duration corresponding to each material image as a video generation guidance parameter to a preset video generation model; inputs the material set into the video generation model and generates the target video based on the video generation guidance parameters. Applying the video generation method described in this application to medical experiment or teaching video generation scenarios can assist medical education and research, improving the way guidance videos are generated in medical education and research.

[0153] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0154] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. A method of video generation, the method comprising: Includes the following steps: Obtain a set of materials for video generation, wherein the set of materials includes material images, text embedding data corresponding to each material image, and descriptive speech data; All material images in the material set are sorted according to a preset sorting rule, and a sequence number is set for all material images based on the sorting results. Based on the sequence number, the image material, the embedded text data, and the descriptive speech data are mapped and organized to obtain the mapping and organization result; Based on the mapping and processing results, the text embedding data that needs to be added to each material image during the video generation process, as well as the duration of the descriptive speech corresponding to each material image, are identified. The duration of the accompanying audio description for each source image is used as a guiding parameter for video generation and added to the preset video generation model. Specifically, this includes: Based on the duration of the descriptive audio corresponding to each image, the video duration corresponding to each image is determined after image diffusion processing using the image diffusion generation method. The total audio duration is obtained by summing the durations of the descriptive audio corresponding to all the images in the material set, and the total audio duration is used as the total video duration of the target video generated after processing by the video generation model. The video duration corresponding to each source image and the total video duration are used as video generation guidance parameters and deployed into the video generation model; The source material set is input into the video generation model, and the target video is generated by combining the video generation guidance parameters.

2. The video generation method according to claim 1, characterized in that, The step of obtaining the material set for video generation specifically includes: The photographic images obtained from medical scientific experiments and used for medical guidance and teaching are used as the source images. The shooting mode selected during the shooting includes adding shooting time information to each photographic image. The introductory text content added for each photographic image is obtained as the text embedding data; The introductory language content recorded for each photographic image is obtained as the descriptive speech data.

3. The video generation method according to claim 2, characterized in that, The step of sorting all material images in the material set according to a preset sorting rule, and setting a sequence number for all material images based on the sorting result, specifically includes: The media images in the media set are sorted based on the shooting time information corresponding to each photographic image. The preset sorting rules include sorting the photographic images according to the order of shooting time information. Based on the sorting results, the position information of each of the source images in the sorting sequence is identified, and the position information is used as the sequence number.

4. The video generation method according to claim 1, characterized in that, The step of identifying the text embedding data that needs to be added to each source image during video generation based on the mapping and processing results, and the corresponding descriptive speech duration for each source image, specifically includes: Based on the mapping and processing results, the text embedding data corresponding to each material image is identified; The text embedding data is set as the text embedding data that needs to be added to the corresponding source image during the video generation process; Based on the mapping and processing results, the descriptive speech data corresponding to each source image is identified; Set the duration of the descriptive voice data to the duration of the descriptive voice for the corresponding source image.

5. The video generation method according to claim 1, characterized in that, The video generation model includes an image diffusion-based video generation model, which comprises a transformer encoding / decoding structure and a U-Net intermediate layer structure. The step of inputting the source set into the video generation model and generating the target video in conjunction with the video generation guidance parameters specifically includes: The encoding layer in the transformer codec structure encodes the material images, text embedding data, and descriptive speech data in the material set, respectively, to obtain the corresponding image encoding features, text encoding features, and speech encoding features. According to the sequence number, the previous condition image and the next condition image are obtained from the source images in sequence to form N-1 groups of images. The sequence number of the previous condition image is 1 less than the sequence number of the next condition image. N is a positive integer, representing the total number of source images in the source set. Based on the different groups of images, N-1 groups of image coding features are obtained. Each group of images includes two source images with a sequence number difference of 1. Each group of image coding features includes the image coding features corresponding to the two source images respectively. By combining a preset time step, the N-1 groups of image coding features are embedded with noise features in an iterative manner to obtain the image coding features with embedded noise features corresponding to the N-1 groups of image coding features respectively. The image coding features embedded with noise features are processed by an attention mechanism and then mapped to the intermediate layer structure of the U-Net for image coding feature denoising to obtain the denoised image coding features. The denoised image encoding features, text encoding features, and speech encoding features are input into the decoding layer of the transformer encoding / decoding structure. Combined with the video generation guidance parameters and the image-text-speech mapping relationship, the target video is generated.

6. The video generation method according to claim 5, characterized in that, The step of generating the target video by combining the video generation guidance parameters and the image-text-speech mapping relationship specifically includes: The display duration of each video image in the target video is controlled according to the video generation guidance parameters; Based on the image-text-speech mapping relationship, the text data output from the decoding layer is added to the reserved text addition position in the corresponding image; and, Based on the image-text-speech mapping relationship, the speech data output by the decoding layer is added to the display time period corresponding to the image.

7. A video generation apparatus, characterized in that, include: The material set acquisition module is used to acquire a material set for video generation, wherein the material set includes material images, text embedding data corresponding to each material image, and descriptive speech data; The image serialization processing module is used to sort all material images in the material set according to a preset sorting rule, and set a sequence number for all material images based on the sorting results; The mapping and processing module is used to perform image-text-speech mapping and processing on the material image, the text embedding data, and the descriptive speech data based on the sequence number, and obtain the mapping and processing result; The mapping relationship recognition module is used to identify, based on the mapping and sorting results, the text embedding data that needs to be added to each material image during the video generation process, and the duration of the descriptive speech corresponding to each material image; The video generation model parameter adding module is used to add the duration of the descriptive audio corresponding to each source image as a video generation guidance parameter to the preset video generation model. Specifically, it includes: Based on the duration of the descriptive audio corresponding to each image, the video duration corresponding to each image is determined after image diffusion processing using the image diffusion generation method. The total audio duration is obtained by summing the durations of the descriptive audio corresponding to all the images in the material set, and the total audio duration is used as the total video duration of the target video generated after processing by the video generation model. The video duration corresponding to each source image and the total video duration are used as video generation guidance parameters and deployed into the video generation model; The target video generation module is used to input the material set into the video generation model and generate the target video in combination with the video generation guidance parameters.

8. A computer device, characterized in that, The method includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the video generation method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the video generation method as described in any one of claims 1 to 6.

Images