Video generation method, device and equipment and storage medium thereof

Abstract

The embodiment of the invention belongs to the technical field of image processing, is applied to scene generation of a video through an image, and relates to a video generation method, device and equipment and a storage medium thereof. Performing image-text-voice mapping arrangement on the material image, the text embedded data and the description voice data; recognizing text embedded data needing to be added in the video generation process of each material image and description voice duration corresponding to each material image; taking the description voice duration as a video generation guidance parameter, and adding the description voice duration into a preset video generation model; and inputting the material set into the video generation model, and generating a target video in combination with the video generation guidance parameters. When the video generation method is applied to a medical experiment or teaching video generation scene, medical education and scientific research can be assisted, and a guidance video generation mode during medical education and scientific research is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and is applied to a video generation scenario using images, and in particular to a video generation method, apparatus, device, and storage medium thereof. Background Art

[0002] With the rapid development of deep learning technology, diffusion models have made significant progress in the field of video generation. Such models can create more coherent and natural video content by generating video sequences frame by frame. By introducing long-term dependencies, diffusion models can capture more complex spatiotemporal relationships in videos, making the generated videos more narrative and emotional.

[0003] In the medical field, the application of video generation technology is of great significance. For example, in the medical field, a large amount of training and educational resources are needed. Video generation technology can be used to create realistic medical case demonstration videos, providing more vivid and intuitive learning resources for medical students and practitioners. How to generate guidance videos for medical education and scientific research has become an urgent problem to be solved. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose a video generation method, device, equipment and storage medium thereof to solve the problem of how to generate guidance videos for medical education and scientific research.

[0005] In order to solve the above technical problems, the embodiment of the present application provides a video generation method, which adopts the following technical solution:

[0006] A video generation method comprises the following steps:

[0007] Acquire a material set for video generation, wherein the material set includes material images, text embedding data corresponding to each material image, and description voice data;

[0008] Sorting all the material images in the material collection according to a preset sorting rule, and setting sequence numbers for all the material images based on the sorting result;

[0009] Based on the sequence number, the material image, the text embedded data and the description voice data are mapped and sorted among image, text and voice to obtain a mapping and sorting result;

[0010] According to the mapping and sorting results, identifying the text embedding data that needs to be added to each material image during the video generation process, and the description voice duration corresponding to each material image;

[0011] The description voice duration corresponding to each material image is used as a video generation guidance parameter and added to the preset video generation model;

[0012] The material set is input into the video generation model, and the target video is generated in combination with the video generation guidance parameters.

[0013] Furthermore, the step of obtaining a material set for video generation specifically includes:

[0014] By shooting, photographic images used for medical guidance teaching are obtained from the medical science experiment process as the material images, wherein the shooting mode selected during shooting includes adding shooting time information to each photographic image;

[0015] obtaining the introductory text content added to each photographic image as the text embedding data;

[0016] The introductory language content recorded respectively for each photographic image is obtained as the descriptive voice data.

[0017] Furthermore, the step of sorting all the material images in the material set according to a preset sorting rule, and setting sequence numbers for all the material images based on the sorting result, specifically includes:

[0018] Sorting all the material images in the material collection based on the shooting time information corresponding to each photographic image, wherein the preset sorting rule includes sorting the photographic images according to the order of the shooting time information;

[0019] In combination with the sorting processing result, the position information of all the material images in the sorting sequence is identified, and the position information is used as the sequence number.

[0020] Furthermore, the step of identifying the text embedding data that needs to be added to each material image during the video generation process and the description voice duration corresponding to each material image according to the mapping and sorting results specifically includes:

[0021] According to the mapping and sorting results, identifying the text embedded data corresponding to each material image;

[0022] The text embedding data is set as the text embedding data that needs to be added to the corresponding material image during the video generation process;

[0023] According to the mapping and sorting results, identifying the description voice data corresponding to each material image;

[0024] The duration of the description voice data is set to the description voice duration of the corresponding material image.

[0025] Furthermore, the step of adding the description voice duration corresponding to each material image as a video generation guidance parameter to a preset video generation model specifically includes:

[0026] According to the description speech duration corresponding to each material image, determine the video duration corresponding to each material image after performing image diffusion processing on each material image using the image diffusion generation method;

[0027] Accumulate the description speech durations corresponding to all the material images in the material set to obtain a total speech duration, and use the total speech duration as the total video duration of the target video generated after being processed by the video generation model;

[0028] The video duration corresponding to each material image and the total video duration are used as video generation guidance parameters and deployed in the video generation model.

[0029] Furthermore, the video generation model includes a video generation model based on image diffusion generation, the video generation model based on image diffusion generation includes a transformer codec structure and a U-Net intermediate layer structure, and the step of inputting the material set into the video generation model and generating a target video in combination with the video generation guidance parameters specifically includes:

[0030] Through the coding layer in the transformer coding and decoding structure, the material images, text embedded data and description voice data in the material set are respectively coded and processed to obtain corresponding image coding features, text coding features and voice coding features;

[0031] sequentially acquiring a previous condition image and a next condition image from the material images according to the sequence number to form N-1 groups of images, wherein the sequence number of the previous condition image is 1 less than the sequence number of the next condition image, and N is a positive integer representing the total number of material images in the material set;

[0032] According to the different groups of images, N-1 groups of image coding features are obtained, wherein one group of images includes two source images whose sequence number difference is 1, and each group of image coding features includes image coding features corresponding to the two source images respectively;

[0033] In combination with a preset time step, the N-1 groups of image coding features are respectively subjected to noise feature embedding in an iterative manner to obtain image coding features of the embedded noise features corresponding to the N-1 groups of image coding features;

[0034] The image coding features embedded with noise features are mapped to the U-Net intermediate layer structure after being processed by the attention mechanism, and the image coding features are denoised to obtain the denoised image coding features;

[0035] The denoised image coding features, text coding features and speech coding features are input into the decoding layer in the transformer codec structure, and the target video is generated by combining the video generation guidance parameters and the image-text-speech mapping relationship.

[0036] Furthermore, the step of generating the target video by combining the video generation guidance parameters and the image-text-speech mapping relationship specifically includes:

[0037] Controlling the display duration of each video image in the target video according to the video generation guidance parameter;

[0038] According to the image-text-speech mapping relationship, the text data output by the decoding layer is added to the reserved text adding position of the corresponding image; and

[0039] According to the image-text-speech mapping relationship, the speech data output by the decoding layer is added to the presentation time period corresponding to the corresponding image.

[0040] In order to solve the above technical problems, the embodiment of the present application also provides a video generation device, which adopts the following technical solution:

[0041] A video generating device, comprising:

[0042] A material set acquisition module, used to acquire a material set for video generation, wherein the material set includes material images, text embedded data corresponding to each material image, and description voice data;

[0043] An image serialization processing module, used to sort all the material images in the material set according to a preset sorting rule, and set sequence numbers for all the material images based on the sorting result;

[0044] A mapping arrangement module, used for performing image-text-speech mapping arrangement on the material image, the text embedded data and the description voice data based on the sequence number to obtain a mapping arrangement result;

[0045] A mapping relationship recognition module, used to identify the text embedding data that needs to be added to each material image during the video generation process, and the description voice duration corresponding to each material image, according to the mapping arrangement result;

[0046] A video generation model parameter adding module is used to add the description voice duration corresponding to each material image as a video generation guidance parameter to a preset video generation model;

[0047] The target video generation module is used to input the material set into the video generation model and generate a target video in combination with the video generation guidance parameters.

[0048] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0049] A computer device comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the above-mentioned video generation method when executing the computer-readable instructions.

[0050] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0051] A computer-readable storage medium stores computer-readable instructions, which implement the steps of the video generation method as described above when executed by a processor.

[0052] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0053] The video generation method described in the embodiment of the present application is obtained by obtaining a material set for video generation; sorting all material images in the material set and setting a sequence number; mapping and arranging the material images, text embedded data and descriptive voice data among images, texts and voices based on the sequence number; identifying the text embedded data that needs to be added to each material image during the video generation process and the descriptive voice duration corresponding to each material image according to the mapping and arranging results; adding the descriptive voice duration corresponding to each material image as a video generation guidance parameter to a preset video generation model; inputting the material set into the video generation model, and generating the target video in combination with the video generation guidance parameter. The video generation method described in the present application is applied to medical experiments or teaching video generation scenarios, which can assist medical education and scientific research, and improve the guidance video generation method during medical education and scientific research. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0055] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0056] Figure 2 is a flow chart of an embodiment of a video generation method according to the present application;

[0057] Figure 3 yes Figure 2 A flowchart of a specific embodiment of step 201 is shown;

[0058] Figure 4 yes Figure 2 A flowchart of a specific embodiment of step 202 is shown;

[0059] Figure 5 yes Figure 2 A flowchart of a specific embodiment of step 204 is shown;

[0060] Figure 6 yes Figure 2 A flowchart of a specific embodiment of step 205 is shown;

[0061] Figure 7 yes Figure 2 A flowchart of a specific embodiment of step 206 is shown;

[0062] Figure 8 yes Figure 7 A flowchart of a specific embodiment of step 706 is shown;

[0063] Fig. 9 is a structural schematic diagram of an embodiment of a video generating device according to the present application;

[0064] Fig.10 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0065] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0066] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0067] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0068] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0069] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0070] The terminal device 101 can be various electronic devices with a display screen and supporting web browsing. In addition to the laptop computer 1011, the tablet computer 1012 or the mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV) player, a laptop computer and a desktop computer, etc.

[0071] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0072] It should be noted that the video generating method provided in the embodiment of the present application is generally executed by a terminal device, and accordingly, the video generating device is generally arranged in the terminal device.

[0073] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0074] Continue to refer Figure 2 , shows a flow chart of an embodiment of a video generation method according to the present application. The video generation method comprises the following steps:

[0075] Step 201, obtaining a material set for video generation, wherein the material set includes material images, text embedding data corresponding to each material image, and description voice data;

[0076] Specifically, the material images include medical education images and / or medical research images, for example: experimental operation images obtained according to the various operation steps of the PCR experiment, the text embedded data includes the experimental operation points, reagent addition amount, operation precautions and reaction mechanism description corresponding to each experimental operation image, and the descriptive voice data includes simple experimental step introduction voice, etc.

[0077] By obtaining a material set for video generation, subsequent video generation is facilitated. Applying the video generation method described in this application to medical experiments or teaching video generation scenarios can assist medical education and scientific research, and improve the guidance video generation method for medical education and scientific research.

[0078] Step 202, sorting all the material images in the material set according to a preset sorting rule, and setting sequence numbers for all the material images based on the sorting result;

[0079] By sorting all the material images in the material set and setting sequence numbers for all the material images based on the sorting results, the target video finally generated follows specific steps. It is assumed that the material images are experimental operation images obtained according to the various operation steps of the PCR experiment. The purpose of sorting is to ensure the correctness of the steps of the generated medical experiment or teaching video.

[0080] Step 203, mapping and arranging the material image, the text embedded data and the description voice data among image, text and voice based on the sequence number to obtain a mapping and arranging result;

[0081] The material image, the text embedded data and the descriptive voice data are mapped and sorted among image, text and voice through the serial numbering, and the target video is generated by combining the text description and the voice description, so that the finally generated target video can be used for teaching demonstration.

[0082] Step 204, identifying the text embedding data that needs to be added to each material image during the video generation process and the description voice duration corresponding to each material image according to the mapping and sorting results;

[0083] Specifically, by identifying the description voice duration corresponding to each material image, the purpose is to control the display duration of each material image in the final generated target video, thereby avoiding the mismatch between image display and voice description.

[0084] Step 205, taking the description voice duration corresponding to each material image as a video generation guidance parameter and adding it to a preset video generation model;

[0085] By taking the description voice duration corresponding to each material image as a video generation guidance parameter and adding it to the preset video generation model, its purpose is to control the display duration of each material image in the final generated target video by using a model generation method, thereby avoiding the mismatch between image display and voice description.

[0086] Step 206: input the material set into the video generation model, and generate a target video in combination with the video generation guidance parameters.

[0087] In this embodiment, by obtaining a material set for video generation; sorting all material images in the material set and setting a sequence number; mapping and arranging the material images, text embedded data and descriptive voice data among images, texts and voices based on the sequence number; identifying the text embedded data that needs to be added to each material image during the video generation process and the descriptive voice duration corresponding to each material image according to the mapping and arranging result; adding the descriptive voice duration corresponding to each material image as a video generation guidance parameter to a preset video generation model; inputting the material set into the video generation model, and generating the target video in combination with the video generation guidance parameter. The video generation method described in this application is applied to medical experiments or teaching video generation scenarios, which can assist medical education and scientific research, and improve the guidance video generation method in medical education and scientific research.

[0088] Continue to refer Figure 3 , in some optional implementations, Figure 3 yes Figure 2 The flowchart of a specific embodiment of step 201 shown includes the following steps:

[0089] Step 301, obtaining photographic images for medical guidance teaching from a medical science experiment process as the material images by shooting, wherein the shooting mode selected during shooting includes adding shooting time information to each photographic image;

[0090] The shooting mode selected during shooting is set to add shooting time information to each photographic image, so as to facilitate the subsequent sorting of each material image.

[0091] Step 302, obtaining the introductory text content added to each photographic image as the text embedding data;

[0092] By obtaining the introductory text content added for each photographic image as the text embedding data, textual description content is added to the corresponding image in the video when the video is generated.

[0093] Step 303: Obtain the introductory language content recorded for each photographic image as the descriptive voice data.

[0094] By obtaining the introductory language content recorded for each photographic image, it is convenient to add descriptive voice to each image in the subsequently generated video, and it is also convenient to control the display time of each image in the video according to the descriptive voice data.

[0095] Continue to refer Figure 4 , in some optional implementations, Figure 4 yes Figure 2 The flowchart of a specific embodiment of step 202 shown includes the following steps:

[0096] Step 401, sorting all the material images in the material collection based on the shooting time information corresponding to each photographic image, wherein the preset sorting rule includes sorting the photographic images according to the order of the shooting time information;

[0097] Step 402: In combination with the sorting processing result, position information of all the material images in the sorting sequence is identified, and the position information is used as the sequence number.

[0098] By sorting all the material images in the material set and setting sequence numbers for all the material images based on the sorting results, the final generated target video follows specific steps, ensuring the correctness of the steps of the generated medical experiment or teaching video.

[0099] Continue to refer Figure 5 , in some optional implementations, Figure 5 yes Figure 2 The flowchart of a specific embodiment of step 204 shown includes the following steps:

[0100] Step 501, identifying text embedded data corresponding to each material image according to the mapping and sorting result;

[0101] Step 502, setting the text embedding data as the text embedding data that needs to be added to the corresponding material image during the video generation process;

[0102] Step 503, identifying the description voice data corresponding to each material image according to the mapping and sorting result;

[0103] Step 504: Set the duration of the description voice data to the description voice duration of the corresponding material image.

[0104] By identifying the text embedding data that needs to be added to each material image during the video generation process, as well as the description voice duration corresponding to each material image, the text embedding data can be added to the video when the target video is subsequently generated, and the total length of the target video and the display length of each image can be controlled in combination with the description voice duration.

[0105] Continue to refer Figure 6 , in some optional implementations, Figure 6 yes Figure 2 The flowchart of a specific embodiment of step 205 shown includes the following steps:

[0106] Step 601, according to the description voice duration corresponding to each material image, determine the video duration corresponding to each material image after image diffusion processing is performed on each material image using an image diffusion generation method;

[0107] Step 602, accumulating the description speech durations corresponding to all the material images in the material set to obtain a total speech duration, and using the total speech duration as the total video duration of the target video generated after being processed by the video generation model;

[0108] Step 603: The video duration corresponding to each material image and the total video duration are used as video generation guidance parameters and deployed into the video generation model.

[0109] By taking the description voice duration corresponding to each material image as a video generation guidance parameter and adding it to the preset video generation model, the generation of the video can be controlled according to the description voice duration, preventing the target video generation time from being too short or too long, thereby ensuring the appropriateness of the duration of the generated target video.

[0110] In this embodiment, the video generation model includes a video generation model based on image diffusion generation, and the video generation model based on image diffusion generation includes a transformer codec structure and a U-Net intermediate layer structure. Specifically, the encoding layer in the transformer codec structure can be directly used as the input layer of the video generation model, and the decoding layer in the transformer codec structure can be used as the output layer of the video generation model. The U-Net intermediate layer structure performs an image diffusion generation step on the material image and is arranged between the encoding layer and the decoding layer.

[0111] Continue to refer Figure 7 , in some optional implementations, Figure 7 yes Figure 2 The flowchart of a specific embodiment of step 206 shown includes the following steps:

[0112] Step 701, encoding the material images, text embedded data and description voice data in the material set respectively through the encoding layer in the transformer encoding and decoding structure to obtain corresponding image encoding features, text encoding features and voice encoding features;

[0113] The corresponding image coding features, text coding features and speech coding features are obtained through the coding layer, so as to facilitate the subsequent decoding and integration of the image coding features, text coding features and speech coding features.

[0114] Step 702, sequentially acquiring a previous condition image and a next condition image from the material images according to the sequence number to form N-1 groups of images, wherein the sequence number of the previous condition image is 1 less than the sequence number of the next condition image, and N is a positive integer representing the total number of material images in the material set;

[0115] Specifically, for example: according to the experimental operation images obtained according to the various operation steps of the PCR experiment, from the beginning of the experiment to the completion of the experiment, a total of N=15 images were taken, and the sequence numbers were 1 to 15 respectively. Then, the image with sequence number 1 is first used as the previous condition image, and the image with sequence number 2 is used as the next condition image to generate the first group of images. Then, the image with sequence number 2 is used as the previous condition image, and the image with sequence number 3 is used as the next condition image to generate the second group of images, and so on, until the last group of images with sequence number 14 as the previous condition image and sequence number 15 as the next condition image is obtained, then a total of N-1, that is, 14 groups of images are obtained.

[0116] By constructing group images, it is possible to introduce the changes from the previous image to the next image as much as possible according to the group image and image diffusion generation method when generating the target video, so that the generated target video pays more attention to the changes between images.

[0117] Step 703, obtaining N-1 groups of image coding features according to the different groups of images, wherein one group of images includes two material images whose sequence number difference is 1, and each group of image coding features includes image coding features corresponding to the two material images respectively;

[0118] Step 704, in combination with a preset time step, the N-1 groups of image coding features are respectively subjected to noise feature embedding in an iterative manner to obtain image coding features with embedded noise features corresponding to the N-1 groups of image coding features;

[0119] Specifically, since the number of material images for generating a video is certain, an image diffusion generation method is adopted. By introducing noise features in an iterative manner according to a preset time step, image coding features embedded with noise features are obtained. Assuming that for the material image with a sequence number of 2, the time step is 0.2 seconds, and the total number of iterations is 60, a total of 60 image coding features embedded with noise features are obtained, and each iteration reintroduces noise features on the basis of the previous iteration.

[0120] Here, the product of the time step and the number of iterations must not exceed the duration of the description voice corresponding to the corresponding material image. That is, assuming that the description voice duration corresponding to the material image with sequence number 3 is 20 seconds and the time step is 0.2 seconds, the maximum number of iterations is 100. This avoids the situation where the image generated by the amplification far exceeds the reserved time for video display when the image is diffused and generated.

[0121] Step 705, mapping the image coding features embedded with the noise features to the U-Net intermediate layer structure after being processed by the attention mechanism, performing image coding feature denoising processing to obtain denoised image coding features;

[0122] Specifically, since the number of material images for generating a video is fixed, an image diffusion generation method is adopted. By introducing noise features in an iterative manner according to a preset time step, and then performing denoising processing, it essentially achieves image expansion of the material images, so that the generated target video is generated based on more images.

[0123] Step 706: input the denoised image coding features, text coding features and speech coding features into the decoding layer in the transformer codec structure, and generate the target video by combining the video generation guidance parameters and the image-text-speech mapping relationship.

[0124] Continue to refer Figure 8 , in some optional implementations, Figure 8 yes Figure 7 The flowchart of a specific embodiment of step 706 shown includes the following steps:

[0125] Step 801, controlling the display duration of each video image in the target video according to the video generation guidance parameter;

[0126] Step 802, adding the text data output by the decoding layer to the reserved text adding position of the corresponding image according to the image-text-speech mapping relationship; and

[0127] Step 803: according to the image-text-speech mapping relationship, the speech data output by the decoding layer is added to the display time period corresponding to the corresponding image.

[0128] Specifically, the display duration of each video image in the target video is controlled according to the video generation guidance parameters, and according to the image-text-voice mapping relationship, the text data output by the decoding layer is added to the text adding position reserved for the corresponding image; and the voice data output by the decoding layer is added to the display time period corresponding to the corresponding image, which not only realizes the display duration control of the target video generation, but also realizes the mimetic processing of the static display image, and adds text and voice description content.

[0129] This application obtains a material set for video generation; sorts all material images in the material set and sets a sequence number; maps and organizes the material images, text embedded data and descriptive voice data among images, texts and voices based on the sequence number; identifies the text embedded data that needs to be added to each material image during the video generation process, and the descriptive voice duration corresponding to each material image according to the mapping and organization results; adds the descriptive voice duration corresponding to each material image as a video generation guidance parameter to a preset video generation model; inputs the material set into the video generation model, and generates a target video in combination with the video generation guidance parameters. The video generation method described in this application is applied to medical experiments or teaching video generation scenarios, which can assist medical education and scientific research, and improve the guidance video generation method for medical education and scientific research.

[0130] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0131] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0132] In the embodiment of the present application, by obtaining a material set for video generation; sorting all material images in the material set and setting a sequence number; mapping and arranging the material images, text embedded data and descriptive voice data among images, texts and voices based on the sequence number; identifying the text embedded data that needs to be added to each material image during the video generation process and the descriptive voice duration corresponding to each material image according to the mapping and arranging result; adding the descriptive voice duration corresponding to each material image as a video generation guidance parameter to a preset video generation model; inputting the material set into the video generation model, and generating the target video in combination with the video generation guidance parameter. The video generation method described in the present application is applied to medical experiments or teaching video generation scenarios, which can assist medical education and scientific research, and improve the guidance video generation method in medical education and scientific research.

[0133] Further references Fig. 9 , as a response to the above Figure 2 The present application provides an embodiment of a video generation device, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0134] like Fig. 9 As shown, the video generation device 900 described in this embodiment includes: a material set acquisition module 901, an image serialization processing module 902, a mapping arrangement module 903, a mapping relationship identification module 904, a video generation model parameter addition module 905 and a target video generation module 906. Among them:

[0135] A material set acquisition module 901 is used to acquire a material set for video generation, wherein the material set includes material images, text embedded data corresponding to each material image, and description voice data;

[0136] An image serialization processing module 902 is used to sort all the material images in the material set according to a preset sorting rule, and set sequence numbers for all the material images based on the sorting result;

[0137] A mapping arrangement module 903 is used to perform image-text-speech mapping arrangement on the material image, the text embedded data and the description voice data based on the sequence number to obtain a mapping arrangement result;

[0138] A mapping relationship identification module 904 is used to identify the text embedding data that needs to be added to each material image during the video generation process and the description voice duration corresponding to each material image according to the mapping arrangement result;

[0139] The video generation model parameter adding module 905 is used to add the description voice duration corresponding to each material image as a video generation guidance parameter to the preset video generation model;

[0140] The target video generation module 906 is used to input the material set into the video generation model and generate a target video in combination with the video generation guidance parameters.

[0141] This application obtains a material set for video generation; sorts all material images in the material set and sets a sequence number; maps and organizes the material images, text embedded data and descriptive voice data among images, texts and voices based on the sequence number; identifies the text embedded data that needs to be added to each material image during the video generation process, and the descriptive voice duration corresponding to each material image according to the mapping and organization results; adds the descriptive voice duration corresponding to each material image as a video generation guidance parameter to a preset video generation model; inputs the material set into the video generation model, and generates a target video in combination with the video generation guidance parameters. The video generation method described in this application is applied to medical experiments or teaching video generation scenarios, which can assist medical education and scientific research, and improve the guidance video generation method for medical education and scientific research.

[0142] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0143] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0144] To solve the above technical problems, the present application also provides a computer device. Fig.10 , Fig.10 This is a basic structural block diagram of the computer device in this embodiment.

[0145] The computer device 10 includes a memory 10a, a processor 10b, and a network interface 10c which are interconnected and communicated via a system bus. It should be noted that: Fig.10 Only a computer device 10 having a component memory 10a, a processor 10b, and a network interface 10c is shown, but it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital processor (DSP), an embedded device, etc.

[0146] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0147] The memory 10a includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 10a can be an internal storage unit of the computer device 10, such as a hard disk or memory of the computer device 10. In other embodiments, the memory 10a can also be an external storage device of the computer device 10, such as a plug-in hard disk equipped on the computer device 10, a smart memory card (Smart Med ia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 10a can also include both the internal storage unit of the computer device 10 and its external storage device. In this embodiment, the memory 10a is generally used to store an operating system and various application software installed on the computer device 10, such as a computer-readable instruction of a video generation method, etc. In addition, the memory 10a can also be used to temporarily store various data that have been output or will be output.

[0148] The processor 10b may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 10b is generally used to control the overall operation of the computer device 10. In the present embodiment, the processor 10b is used to run the computer-readable instructions or process data stored in the memory 10a, such as the computer-readable instructions for running the video generation method.

[0149] The network interface 10c may include a wireless network interface or a wired network interface. The network interface 10c is generally used to establish a communication connection between the computer device 10 and other electronic devices.

[0150] The computer device proposed in this embodiment belongs to the field of image processing technology and is applied to the scene of video generation using images. This application obtains a material set for video generation; sorts all material images in the material set and sets a sequence number; maps and organizes the material images, text embedded data and descriptive voice data among images, texts and voices based on the sequence number; identifies the text embedded data that needs to be added to each material image during the video generation process and the descriptive voice duration corresponding to each material image according to the mapping and organization results; adds the descriptive voice duration corresponding to each material image as a video generation guidance parameter to the preset video generation model; inputs the material set into the video generation model, and generates the target video in combination with the video generation guidance parameter. The video generation method described in this application is applied to medical experiments or teaching video generation scenarios, which can assist medical education and scientific research, and improve the guidance video generation method in medical education and scientific research.

[0151] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by a processor to enable the processor to perform the steps of the video generation method as described above.

[0152] The computer-readable storage medium proposed in this embodiment belongs to the field of image processing technology and is applied to the video generation scenario using images. This application obtains a material set for video generation; sorts all the material images in the material set and sets a sequence number; maps and organizes the material images, text embedded data and descriptive voice data between images, texts and voices based on the sequence number; identifies the text embedded data that needs to be added to each material image during the video generation process and the descriptive voice duration corresponding to each material image according to the mapping and organization results; adds the descriptive voice duration corresponding to each material image as a video generation guidance parameter to a preset video generation model; inputs the material set into the video generation model, and generates the target video in combination with the video generation guidance parameters. The video generation method described in this application is applied to medical experiments or teaching video generation scenarios, which can assist medical education and scientific research, and improve the guidance video generation method for medical education and scientific research.

[0153] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0154] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A video generation method, characterized in that: The steps include: Acquire a material set for video generation, wherein the material set includes material images, text embedding data corresponding to each material image, and description voice data; Sorting all the material images in the material collection according to a preset sorting rule, and setting sequence numbers for all the material images based on the sorting result; Based on the sequence number, the material image, the text embedded data and the description voice data are mapped and sorted among image, text and voice to obtain a mapping and sorting result; According to the mapping and sorting results, identifying the text embedding data that needs to be added to each material image during the video generation process, and the description voice duration corresponding to each material image; The description voice duration corresponding to each material image is used as a video generation guidance parameter and added to the preset video generation model; The material set is input into the video generation model, and the target video is generated in combination with the video generation guidance parameters.

2. The video generation method according to claim 1, characterized in that: The step of obtaining a material set for video generation specifically includes: By shooting, photographic images used for medical guidance teaching are obtained from the medical science experiment process as the material images, wherein the shooting mode selected during shooting includes adding shooting time information to each photographic image; obtaining the introductory text content added to each photographic image as the text embedding data; The introductory language content recorded respectively for each photographic image is obtained as the descriptive voice data.

3. The video generation method according to claim 2, characterized in that: The step of sorting all the material images in the material set according to a preset sorting rule, and setting sequence numbers for all the material images based on the sorting result, specifically includes: Sorting all the material images in the material collection based on the shooting time information corresponding to each photographic image, wherein the preset sorting rule includes sorting the photographic images according to the order of the shooting time information; In combination with the sorting processing result, the position information of all the material images in the sorting sequence is identified, and the position information is used as the sequence number.

4. The video generation method according to claim 1, characterized in that: The step of identifying the text embedding data that needs to be added to each material image during the video generation process and the description voice duration corresponding to each material image according to the mapping and sorting results specifically includes: According to the mapping and sorting results, identifying the text embedded data corresponding to each material image; The text embedding data is set as the text embedding data that needs to be added to the corresponding material image during the video generation process; According to the mapping and sorting results, identifying the description voice data corresponding to each material image; The duration of the description voice data is set to the description voice duration of the corresponding material image.

5. The video generation method according to claim 1 or 4, characterized in that: The step of adding the description voice duration corresponding to each material image as a video generation guidance parameter to a preset video generation model specifically includes: According to the description speech duration corresponding to each material image, determine the video duration corresponding to each material image after performing image diffusion processing on each material image using the image diffusion generation method; Accumulate the description speech durations corresponding to all the material images in the material set to obtain a total speech duration, and use the total speech duration as the total video duration of the target video generated after being processed by the video generation model; The video duration corresponding to each material image and the total video duration are used as video generation guidance parameters and deployed in the video generation model.

6. The video generation method according to claim 1, characterized in that: The video generation model includes a video generation model based on image diffusion generation, the video generation model based on image diffusion generation includes a transformer codec structure and a U-Net intermediate layer structure, and the step of inputting the material set into the video generation model and generating a target video in combination with the video generation guidance parameters specifically includes: Through the coding layer in the transformer coding and decoding structure, the material images, text embedded data and description voice data in the material set are respectively coded and processed to obtain corresponding image coding features, text coding features and voice coding features; sequentially acquiring a previous condition image and a next condition image from the material images according to the sequence number to form N-1 groups of images, wherein the sequence number of the previous condition image is 1 less than the sequence number of the next condition image, and N is a positive integer representing the total number of material images in the material set; According to the different groups of images, N-1 groups of image coding features are obtained, wherein one group of images includes two source images whose sequence number difference is 1, and each group of image coding features includes image coding features corresponding to the two source images respectively; In combination with a preset time step, the N-1 groups of image coding features are respectively subjected to noise feature embedding in an iterative manner to obtain image coding features of the embedded noise features corresponding to the N-1 groups of image coding features; The image coding features embedded with noise features are mapped to the U-Net intermediate layer structure after being processed by the attention mechanism, and the image coding features are denoised to obtain the denoised image coding features; The denoised image coding features, text coding features and speech coding features are input into the decoding layer in the transformer codec structure, and the target video is generated by combining the video generation guidance parameters and the image-text-speech mapping relationship.

7. The video generation method according to claim 6, characterized in that: The step of generating the target video by combining the video generation guidance parameters and the image-text-speech mapping relationship specifically includes: Controlling the display duration of each video image in the target video according to the video generation guidance parameter; According to the image-text-speech mapping relationship, the text data output by the decoding layer is added to the reserved text adding position of the corresponding image; and According to the image-text-speech mapping relationship, the speech data output by the decoding layer is added to the presentation time period corresponding to the corresponding image.

8. A video generating device, characterized in that: include: A material set acquisition module, used to acquire a material set for video generation, wherein the material set includes material images, text embedded data corresponding to each material image, and description voice data; An image serialization processing module, used to sort all the material images in the material set according to a preset sorting rule, and set sequence numbers for all the material images based on the sorting result; A mapping arrangement module, used for performing image-text-speech mapping arrangement on the material image, the text embedded data and the description voice data based on the sequence number to obtain a mapping arrangement result; A mapping relationship recognition module, used to identify the text embedding data that needs to be added to each material image during the video generation process, and the description voice duration corresponding to each material image, according to the mapping arrangement result; A video generation model parameter adding module is used to add the description voice duration corresponding to each material image as a video generation guidance parameter to a preset video generation model; The target video generation module is used to input the material set into the video generation model and generate a target video in combination with the video generation guidance parameters.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the video generation method according to any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the video generation method according to any one of claims 1 to 7 are implemented.

Images