Video generation method, readable storage medium, program product and electronic equipment

Abstract

The invention relates to the technical field of image processing, and discloses a video generation method, a readable storage medium, a program product and electronic equipment. The first electronic equipment can execute image-to-video processing on at least part of image materials in the M multimedia materials so as to convert the at least part of image materials into video data, and / or executing video expansion processing on at least part of video materials in the M multimedia materials so as to expand the part of video materials into video data with longer video duration. And then, the first electronic equipment can clip the video data obtained based on the image-to-video processing and / or the video expansion processing and the multimedia materials which are not subjected to the image-to-video processing and the video expansion processing to obtain an editing result. Thus, the at least part of the image materials can be presented in a video mode, and / or the at least part of the video materials can occupy a long time in the editing result, and the visual effect of the editing result can be improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a video generation method, a readable storage medium, a program product, and an electronic device. Background Technology

[0002] To improve video editing efficiency, an increasing number of electronic devices can automatically edit user-selected multimedia materials into videos. In these video editing solutions, electronic devices typically composite image materials and image frames (or video clips) extracted from video materials into a video template to obtain the edited result. This often results in image materials appearing as still images or simple animations (such as scaling, blurring, etc.) in the edited result, or short video materials occupying a small portion of the edited result, leading to poor visual effects and a negative user experience. Summary of the Invention

[0003] In view of this, this application provides a video generation method, a readable storage medium, a program product, and an electronic device, which are beneficial to improving the visual effect of video editing results and thus enhancing the user experience.

[0004] A first aspect provides a video generation method applied to a first electronic device. The method includes: receiving a first instruction, wherein the first instruction instructs M multimedia materials to be edited into a video, the M multimedia materials including at least one image material and / or at least one video material, and M being an integer greater than 1; acquiring N video data, wherein the N video data are obtained by performing a first processing on each of the N multimedia materials in the M multimedia materials, and N being an integer greater than or equal to 1 and less than or equal to M; generating a first video based on the N video data and MN multimedia materials in the M multimedia materials, wherein the MN multimedia materials are multimedia materials other than the N multimedia materials in the M multimedia materials; wherein, when the N multimedia materials include a first image material and the N video data include first video data corresponding to the first image material, the first processing includes processing to generate the first video data based on the first image material; and / or, when the N multimedia materials include a first video material and the N video data include second video data corresponding to the first video material, the first processing includes processing to generate second video data based on the first video material, wherein the video duration of the second video data is greater than the video duration of the first video material.

[0005] In the above method, if the above N multimedia materials include R (R is an integer greater than or equal to 1) image materials, the first electronic device generates the first video based on the video data obtained after the first processing of the R image materials and the MN multimedia materials. This allows the R image materials to be presented as video data rather than static images or simple animations, which is beneficial to improving the visual effect of the first video.

[0006] If the aforementioned N multimedia materials include S (S is an integer greater than or equal to 1) video materials, the first electronic device generates a first video based on the video data obtained after performing a first processing on each of the S video materials, and the N multimedia materials. Since the first processing of the video materials increases the video duration of the S video materials, the video data corresponding to these S video materials occupies a longer duration in the first video. This avoids the situation where the content of the video materials cannot be reflected in the first video due to their short duration, thus improving the visual effect of the first video.

[0007] In one possible implementation of the first aspect above, the N multimedia materials include at least one image material that satisfies the first condition, and / or at least one video material that satisfies the second condition.

[0008] In one possible implementation of the first aspect described above, the first condition includes at least one of the following conditions: the image material is an image material selected by the user; at least one subject in the image material is a preset subject; the scene tag in the image material matches the preset scene tag; the image material includes a portrait, and the size of the face in the portrait is greater than or equal to a preset size; the image material includes a portrait, and the degree to which the face in the portrait is occluded is less than or equal to a first degree threshold; the image material includes a portrait, and the degree to which the portrait is occluded is less than or equal to a second degree threshold; the image material includes a portrait, and the clarity of the face in the portrait is greater than or equal to a clarity threshold; the image material includes a portrait, and the shooting angle corresponding to the portrait is within a preset angle range.

[0009] Based on the aforementioned first condition, for image footage containing human figures, the first processing can be performed only on image footage with relatively good human figure quality (e.g., the degree of face occlusion is less than or equal to a first threshold, the degree of face occlusion is less than or equal to a second threshold, the clarity of the face is greater than or equal to a clarity threshold, the shooting angle corresponding to the human figure is within a preset angle range, etc.). This avoids poor video data quality due to poor human figure quality in the image footage. For image footage excluding human figures, the first processing can be performed only on image footage containing a preset scene or a preset subject, avoiding poor video data quality caused by performing the first processing on image footage unsuitable for it.

[0010] In one possible implementation of the first aspect above, the second condition includes at least one of the following conditions: the video material is a video material selected by the user; the video duration of the video material is less than or equal to a preset duration; at least one subject in the video material is a preset subject; and the video quality of the video material is lower than or equal to a video quality threshold.

[0011] In one possible implementation of the first aspect above, the first video data is generated by the first electronic device calling the first model based on the first image material, or the first video data is received by the first electronic device after the first electronic device sends the first image material to the second electronic device; and / or, the second video data is generated by the first electronic device calling the second model based on the first video material, or the second video data is received by the first electronic device after the first electronic device sends the first video material to the second electronic device.

[0012] In this implementation, the video data corresponding to the image material (such as the first image material) among the N multimedia materials is obtained by inputting the first image material into the first model; the video data corresponding to the video material (such as the first video material) among the N multimedia materials is obtained by inputting the first video material into the second model. The first model and the second model can be the same or different.

[0013] In one possible implementation of the first aspect above, the first video data is generated by the first electronic device or the second electronic device by calling the first model based on the first image material and the first prompt information; and / or the second video data is generated by the first electronic device or the second electronic device by calling the second model based on the first video material and the second prompt information.

[0014] In one possible implementation of the first aspect above, the first prompt information includes a first prompt text input by the user and / or a second prompt text generated based on M multimedia materials; and / or, the second prompt information includes a third prompt text input by the user and / or a fourth prompt text generated based on M multimedia materials.

[0015] In this implementation, when the prompt information collected during the generation of the first video data includes user-inputted prompt text (e.g., the first prompt text), the user's personalized needs for the generated video data can be met. When the prompt information collected during the generation of the first video data includes prompt text generated based on M multimedia materials (e.g., the second prompt text), the high relevance between the generated video data and the M multimedia materials can be ensured. When the prompt information collected during the generation of the second video data includes user-inputted prompt text (e.g., the third prompt text), the user's personalized needs for the generated video data can be met. When the prompt information collected during the generation of the second video data includes prompt text generated based on M multimedia materials (e.g., the second prompt text), the high relevance between the generated video data and the M multimedia materials can be ensured.

[0016] In one possible implementation of the first aspect described above, the second prompt text may be obtained by inputting one or more of the subject labels, scene labels, and semantic labels of the first image material (and / or M multimedia materials) into the prompt word enhancement model.

[0017] In one possible implementation of the first aspect above, the fourth cue text may be obtained by inputting one or more of the subject tags, scene tags, and semantic tags of the first video material or (and / or M multimedia materials) into the cue word enhancement model.

[0018] In one possible implementation of the first aspect above, the video duration of the first video data is determined based on the video duration of at least a portion of the video materials in at least one video material, or the video duration of the first video data is a preset duration.

[0019] In one possible implementation of the first aspect above, generating a first video based on N video data and MN multimedia materials from M multimedia materials includes: using a first video template to generate a first video based on N video data and MN multimedia materials.

[0020] In one possible implementation of the first aspect above, the first video template is a video template whose template parameters match at least one of a plurality of video templates with N video data and MN multimedia materials, or a video template selected by the user from a plurality of video templates, wherein the template parameters include at least one of template size, template orientation, template semantic tags, template scene tags, number of material slots, slot duration of material slots, and slot type of material slots.

[0021] In one possible implementation of the first aspect above, the first video template satisfies a third condition, wherein the third condition includes at least one of the following conditions: the template scene tag of the first video template matches the scene tags of M multimedia materials; the number of material slots in the first video template is greater than or equal to M; the template direction of the first video template matches the material direction of the M multimedia materials; the slot duration of the first material slot with the longest slot duration in the material slots of the first video template is less than or equal to the video duration of each video material in at least one video material; the number of slots in the material slots of the first video template whose slot type is video and whose slot duration is less than a first duration is greater than or equal to N, and the first duration is the video duration of each video data in the N video data; the distribution of the slot types of the material slots of the first video template matches the material types of the N video data and the M multimedia materials.

[0022] In this implementation, when determining the first video template, both the scenario of the video template and the slot duration of the media slots in the video template, the video duration of N video data items, and the video duration of the video media items in the MN multimedia media items are considered. This avoids situations where the slot duration of the media slots in the video template does not match the video data, leading to black screens, white screens, or the display of other content in the first video, thus improving the quality of the first video.

[0023] In one possible implementation of the first aspect above, the first video template is the video template whose semantic tags have the highest similarity to the semantic tags of M multimedia materials among one or more video templates that satisfy the third condition.

[0024] In this implementation, when selecting the first video template, the matching between the semantic tags of the video template and the multimedia materials or video data is also considered.

[0025] In one possible implementation of the first aspect described above, the first video template is received by the first electronic device from a third electronic device, or determined by the first electronic device from a plurality of video templates.

[0026] In this implementation, the first video template can be selected by a first electronic device from multiple video templates, or it can be selected by a third electronic device from multiple video templates and then sent to the first electronic device. The third electronic device can be the same as or different from the aforementioned second electronic device. For example, the first electronic device can send the material information of M multimedia materials to the third electronic device, so that the third electronic device can select the first video template based on the material information of the M multimedia materials.

[0027] Secondly, an improved video generation method is provided, comprising: a first electronic device receiving a first instruction, wherein the first instruction instructs to edit M multimedia materials into a video, wherein the M multimedia materials include at least one image material and / or at least one video material, and M is an integer greater than 1; the first electronic device sending N multimedia materials from the M multimedia materials to a second electronic device, wherein N is an integer greater than or equal to 1 and less than or equal to M; the second electronic device performing a first processing on the N multimedia materials respectively to obtain N video data, and sending the N video data to the first electronic device, wherein, in the case where the N multimedia materials include a first image material, the first processing includes processing to generate first video data based on the first image material, and / or, in the case where the N multimedia materials include a first video material, the first processing includes processing to generate second video data based on the first video material, wherein the video duration of the second video data is greater than the video duration of the first video data; the first electronic device generating a first video based on the N video data and MN multimedia materials from the M multimedia materials, wherein the MN multimedia materials are multimedia materials other than the N multimedia materials from the M multimedia materials.

[0028] In the above method, if the above N multimedia materials include R (R is an integer greater than or equal to 1) image materials, the first electronic device generates the first video based on the video data obtained after the first processing of the R image materials and the MN multimedia materials. This allows the R image materials to be presented as video data rather than static images or simple animations, which is beneficial to improving the visual effect of the first video.

[0029] If the aforementioned N multimedia materials include S (S is an integer greater than or equal to 1) video materials, the first electronic device generates a first video based on the video data obtained after performing a first processing on each of the S video materials, and the N multimedia materials. Since the first processing of the video materials increases the video duration of the S video materials, the video data corresponding to these S video materials occupies a longer duration in the first video. This avoids the situation where the content of the video materials cannot be reflected in the first video due to their short duration, thus improving the visual effect of the first video.

[0030] Thirdly, this application provides a readable storage medium including one or more programs that, when executed on an electronic device, enable the electronic device to implement any of the video generation methods provided in the first aspect above.

[0031] Fourthly, this application provides a computer program product that, when run on an electronic device, enables the electronic device to implement any of the video generation methods provided in the first aspect above.

[0032] Fifthly, this application provides an electronic device, comprising: a memory for storing one or more programs; and a processor for executing one or more programs to cause the electronic device to implement any of the video generation methods provided in the first aspect above.

[0033] It should be noted that the beneficial effects of the third to fifth aspects mentioned above can be referred to the beneficial effects described in the first and second aspects, and will not be repeated here. Attached Figure Description

[0034] Figure 1 According to some embodiments of this application, a schematic diagram of an interface for editing video based on user operation is shown on a mobile phone 10.

[0035] Figure 2 According to some embodiments of this application, a schematic diagram of compositing multimedia material into a video template M3 is shown.

[0036] Figure 3A According to some embodiments of this application, a schematic diagram of the interface displayed by a mobile phone 10 after a user clicks the one-click photo option U113 is shown.

[0037] Figure 3B According to some embodiments of this application, a schematic diagram of video data P1-V is shown.

[0038] Figure 3C According to some embodiments of this application, a schematic diagram of video data P2-V is shown.

[0039] Figure 3D According to some embodiments of this application, a schematic diagram of video data P3-V is shown.

[0040] Figure 3E According to some embodiments of this application, a schematic diagram of an interface for previewing video data generated based on image materials is shown.

[0041] Figure 3F According to some embodiments of this application, a schematic diagram of the interface displayed by a mobile phone 10 after a user clicks on the control U136' is shown.

[0042] Figure 3G According to some embodiments of this application, a schematic diagram of compositing multimedia material into a video template M11 is shown.

[0043] Figure 3H According to some embodiments of this application, a schematic diagram of a clipping result interface U17 is shown.

[0044] Figure 4AAccording to some embodiments of this application, a schematic diagram of a preference selection interface U13″ is shown.

[0045] Figure 4B According to some embodiments of this application, a schematic diagram of video data V2' is shown.

[0046] Figure 5 According to some embodiments of this application, a schematic diagram of compositing multimedia material into a video template M21 is shown.

[0047] Figure 6 According to some embodiments of this application, an interactive flowchart of a video generation method is shown.

[0048] Figure 7 According to some embodiments of this application, a flowchart illustrating a process for determining image material suitable for performing image-to-video processing is shown.

[0049] Figure 8 According to some embodiments of this application, a schematic diagram of a process for generating R video data based on R image materials is shown.

[0050] Figure 9 According to some embodiments of this application, a schematic diagram of a process for generating S video data based on S video materials is shown.

[0051] Figure 10 According to some embodiments of this application, a flowchart is shown for determining a first video template that matches M multimedia materials among a plurality of video templates.

[0052] Figure 11 According to some embodiments of this application, a flowchart of a video generation method is shown.

[0053] Figure 12 According to some embodiments of this application, a schematic diagram of the structure of a video generation apparatus 100 is shown.

[0054] Figure 13 A schematic diagram of a software architecture is shown according to some embodiments of this application.

[0055] Figure 14 According to some embodiments of this application, a structural schematic diagram of a mobile phone 10 is shown. Detailed Implementation

[0056] The illustrative embodiments of this application include, but are not limited to, video generation methods, readable storage media, program products, and electronic devices.

[0057] As described in the background section, electronic devices typically composite image materials and image frames (or video clips) extracted from video materials into a video template to obtain the editing result. This often results in image materials appearing as still images or simple animations (such as scaling, blurring, etc.) in the edited result, or short video materials occupying a small portion of the edited result, leading to poor visual effects and negatively impacting the user experience.

[0058] For example, Figure 1 According to some embodiments of this application, a schematic diagram of an interface for editing video based on user operation is shown on a mobile phone 10.

[0059] like Figure 1 As shown, after detecting that the user selects image material P1, image material P2, image material P3, video material V1, video material V2, and video material V3 in the gallery interface U11, and then clicks the creation control U111, the mobile phone 10 displays the menu U112. Menu U112 may include a one-click video creation option U113 and a collage option U114. After detecting that the user clicks the one-click video creation option U113 in menu U112, the mobile phone 10 can display the template selection interface U12. The template selection interface U12 may include options corresponding to video templates (e.g., video templates M1 to M6) suitable for editing the multimedia materials selected by the user. Then, after detecting that the user selects the video template M3, the mobile phone 10 combines image material P1, image material P2, image material P3, video material V1, video material V2, and video material V3 into the video template M3 to obtain the editing result.

[0060] Specifically, Figure 2 According to some embodiments of this application, a schematic diagram of compositing multimedia material into a video template M3 is shown.

[0061] like Figure 2 As shown, video template M3 includes 6 media slots, such as media slots L1 to L6. Among them, media slots L1, L4, and L6 are of video type, while media slots L2, L3, and L5 are of image type.

[0062] It's important to note that a media slot is a container within a video template used to hold multimedia content. Each media slot can hold one multimedia element (such as video footage, image footage, etc.). The video template defines the slot duration, slot type, and the presentation format of the multimedia element within the slot (e.g., animation, transition effects, background image, background music, etc.) for each media slot. The slot duration indicates the total video length occupied by the multimedia element in the edited result. The slot type indicates the type of multimedia element to be incorporated into the media slot, which can include, but is not limited to, images and videos. For ease of description, media slots with a video type will be referred to as video slots, and video slots with an image type will be referred to as image slots.

[0063] After detecting that the user clicks the generate control U121 in the template selection interface U12 after selecting video template M3, mobile phone 10 can composite image material P1, image material P2, and image material P3 into material slots L2, L3, and L5 respectively. It can also composite multiple image frames extracted from video material V1 that occupy the same video duration as the slot duration of material slot L1 into material slot L1, multiple image frames extracted from video material V2 that occupy the same video duration as the slot duration of material slot L4 into material slot L4, and multiple image frames extracted from video material V3 that occupy the same video duration as the slot duration of material slot L6 into material slot L6, thereby obtaining the editing result.

[0064] Based on the above process, the editing results of image materials (such as image material P1, image material P2, and image material P3) are usually just static images, or simple animations such as scaling, moving, and blurring based on the presentation format corresponding to the material slots in the video template, resulting in poor visual effects. Furthermore, since the video length occupied by video materials in the editing result is usually less than the video length of the video material itself, if a video material is short, the video length occupied by that video material in the editing result may be too short to reflect the content of the video material, affecting the user experience.

[0065] To improve the visual effects and user experience of the edited multimedia materials, this application provides a video generation method. In this method, after receiving a first instruction to edit M (M is an integer greater than 1) multimedia materials into a video, a first electronic device can convert at least a portion of the image materials (e.g., the first image material) into video data by performing image-to-video processing on at least a portion of the image materials (as an example of the first processing). This results in at least a portion of the main elements in the converted video data exhibiting dynamic changes (e.g., human / animal activity, vehicle movement, plants / clouds swaying in the wind, water flow in lakes / seas / rivers / waterfalls, etc.). Alternatively, it can perform video augmentation processing on at least a portion of the video materials (e.g., short-duration video materials, such as the first video material) (as an example of the first processing) to augment the video materials into longer-duration video data. Then, the first electronic device can edit the video data obtained from the above-mentioned image-to-video processing and / or video augmentation processing, as well as the multimedia materials from M multimedia materials that have not undergone image-to-video processing and video augmentation processing, to obtain the editing result (denoted as the first video).

[0066] It should be noted that film / video editing involves selecting, combining, adjusting, and refining raw multimedia materials (such as video, audio, image, and text materials) obtained through shooting, recording, generation via artificial intelligence models, or other means, ultimately creating a video with a complete narrative, emotional expression, and visual appeal. During the editing process, electronic devices can select, combine, adjust, and refine raw multimedia materials based on user input, by invoking artificial intelligence models, or by running editing applications or algorithms.

[0067] In this way, at least some of the image materials in the M multimedia materials in the first video can be presented in the form of video, and / or the video materials with shorter video durations among the M multimedia materials can occupy a longer duration in the first video, thereby better presenting the content in the video, which is beneficial to improving the user experience.

[0068] It should be noted that image-to-video processing can be any process that converts static images into video data. For example, it can be achieved through image-to-video generation models such as Open-Sora, stable video diffusion (SVD) models, animated diffusion models, and image-to-video models based on diffusion models, converting images into dynamic video. In the video data obtained through image-to-video processing, at least a portion of the image material exhibits dynamic changes, rather than a simple animation of the entire image material changing.

[0069] It should be noted that video augmentation can be any process that increases the duration of video footage, including but not limited to frame interpolation and generating longer video data based on video footage using video augmentation models. For example, video augmentation models can include, but are not limited to, time-series and image augmentation (MOTIA) models, diffusion models, etc.

[0070] Below, taking the first electronic device as a mobile phone, M multimedia materials are included. Figure 1 Taking image material P1, image material P2, image material P3, video material V1, video material V2, and video material V3 as examples, the technical solution of this application is introduced.

[0071] For example, Figure 3A According to some embodiments of this application, a schematic diagram of the interface displayed by another mobile phone 10 in response to a user clicking the one-click photo option U113 is shown.

[0072] refer to Figure 3A After detecting that the user clicks the "One-Click Video Creation" option U113 in the aforementioned gallery interface U11, the mobile phone 10 can display the preference selection interface U13. The preference selection interface U13 may include a theme option U131, a video length option U132, a material adjustment option U133, an artificial intelligence (AI) image-generated video option U134, an AI video augmentation option U135, and a generation control U136. Among them, the theme option U131 is used to indicate or configure the theme (or scene) of M multimedia materials; the video length option U132 is used to indicate or configure the video length of the edited result to be generated; the material adjustment option U133 is used to add, replace, or delete multimedia materials; the AI ​​image-generated video option U134 is used to configure whether to enable or disable image-generated video processing for at least some of the image materials in N multimedia materials; and the AI ​​video augmentation option U135 is used to configure whether to enable or disable video augmentation processing for at least some of the video materials in N multimedia materials.

[0073] It should be noted that the theme (or scene) of the M multimedia materials can be configured by the user based on the selection of various themes (such as life fragments, travel, parent-child, selfie, scenery, food, cute pets, happy time, etc.) in the theme option U131, or it can be determined by the phone 10 based on the M multimedia materials. There is no limitation here.

[0074] It should be noted that the final video length can be configured by the user based on the various durations selected in the video length option U132 (such as automatic, 0-15 seconds, 15-30 seconds, 30 seconds or more, etc.), or it can be determined by the phone based on M multimedia materials. There is no limitation here.

[0075] In some embodiments, after detecting that the user has turned on the switch of the AI ​​image-generated video option U134, the mobile phone 10 can determine the image material suitable for image-generated video processing from among M multimedia materials, and generate corresponding video data based on the image material suitable for image-generated video processing.

[0076] For example, refer to Figure 3A After detecting that the user clicks the AI ​​Image-to-Video option U134, the mobile phone 10 can determine the image materials suitable for image-to-video processing from among M multimedia materials (e.g., image materials that meet the first condition, such as image material P1, image material P2, and image material P3). Then, the mobile phone 10 can display a preference selection interface U13'. In the preference selection interface U13', the switch for the AI ​​Image-to-Video option U134' is turned on, and the options in the material adjustment options corresponding to the image materials suitable for image-to-video processing include an AI processing identifier to prompt the user that this part of the material is suitable for (or will be used for, or has already been used for) generating video data. For example, the option corresponding to image material P1 in the preference selection interface U13' may include the AI ​​processing identifier U137.

[0077] In some embodiments, after determining image materials suitable for performing image-to-video processing, the mobile phone 10 can generate video data corresponding to each image material.

[0078] For example, refer to Figure 3B The main subject in image material P1 is a person, and the video data P1-V generated based on image material P1 can be the person's arms hanging naturally ( Figure 3B (a) gradually moves to the crossed position in front of the body. Figure 3B (b) in the middle, then gradually move to a hugging position in front of the chest ( Figure 3B The dynamic scene in (c) of the video.

[0079] For example, referring to 3C, the main subject in image material P2 is leaves, and the video data P2-V generated based on image material P2 can be a dynamic picture of leaves and branches moving with the wind.

[0080] For example, in 3D, the main elements in image material P3 include clouds and trees. The video data P3-V generated based on image material P3 can be a dynamic scene where the trees remain stationary and the clouds move from the left to the right of the screen.

[0081] In some embodiments, the mobile phone 10 can display video data generated based on image material P1, image material P2, or image material P3 respectively, based on the user's selection of image material P1, image material P2, or image material P3 in the material adjustment option U133 of the preference selection interface U13'.

[0082] For example, Figure 3E According to some embodiments of this application, a schematic diagram of an interface for previewing video data generated based on image materials is shown.

[0083] When the mobile phone 10 detects that the user clicks on the option corresponding to image material P1 in the material adjustment options of the preference selection interface U13', a preview interface U14 can be displayed. The preview interface U14 can include a preview area U141, video data options U142, a regeneration control U143, a generation control U144, and a return control U145. Among them, the preview area U141 is used to display a preview of the generated video data; the video data options U142 includes options corresponding to video data P1-V, video data P2-V, and video data P3-V; the regeneration control U143 is used to trigger the regeneration of video data based on the image material; the generation control U144 is used to trigger the editing of video data P1-V, video data P2-V, video data P3-V, video material V1, video material V2, and video material V3 to obtain the first video; and the return control U145 is used to trigger a return to the preference selection interface U13'.

[0084] It should be noted that after the phone detects that the user selects a video data option in the video data option U142, the preview area U141 displays the screen corresponding to the video data selected by the user.

[0085] In some embodiments, after detecting that the user clicks the generate control U136 on the preference selection interface U13, or after detecting that the user clicks the generate control U144 on the preview interface U14, the mobile phone 10 can merge the video data P1-V, video data P2-V, video data P3-V, video material V1, video material V2, and video material V3 into the first video template to obtain the editing result of image material P1, image material P2, image material P3 and video material V1, video material V2, and video material V3 (as an example of the first video).

[0086] For example, refer to Figure 3FAfter detecting a user's click on the generation control U136', the mobile phone 10 can display a preference selection interface U15. The preference selection interface U15 may include progress indicator information U151, which can indicate the progress of the mobile phone 10 in acquiring a video template matching the user's selected multimedia material, or the progress of the mobile phone 10 in acquiring video data P1-V, video data P2-V, and video data P3-V. After the mobile phone 10 acquires the first video template matching the user's selected multimedia material, as well as video data P1-V, video data P2-V, and video data P3-V, and video material V1, video material V2, and video material V3, it can combine the video data P1-V, video data P2-V, video data P3-V, video material V1, video material V2, and video material V3 with the first video template and display the combination interface U16. The compositing interface U16 may include progress indication information U161, which is used to indicate the progress of the mobile phone 10 in compositing video data P1-V, video data P2-V, video data P3-V, video material V1, video material V2 and video material V3 with the first video template.

[0087] For example, Figure 3G According to some embodiments of this application, a schematic diagram is shown of fusing video data P1-V, video data P2-V, video data P3-V, video material V1, video material V2, and video material V3 into a video template M11.

[0088] like Figure 3G As shown, the material slots of video template M11 include 6 video slots, such as video slots L7 to L12. Mobile Phone 10 can composite multiple image frames extracted from video material V1 that occupy the same video duration as the slot duration of video slot L7 into video slot L7; it can composite multiple image frames extracted from video data P1-V that occupy the same video duration as the slot duration of video slot L8 into video slot L8; it can composite multiple image frames extracted from video data P2-V that occupy the same video duration as the slot duration of video slot L9 into video slot L9; it can composite multiple image frames extracted from video material V2 that occupy the same video duration as the slot duration of video slot L10 into video slot L10; it can composite multiple image frames extracted from video data P3-V that occupy the same video duration as the slot duration of video slot L11 into video slot L11; and it can composite multiple image frames extracted from video material V3 that occupy the same video duration as the slot duration of video slot L12 into video slot L12, thereby obtaining the editing result.

[0089] Continue to refer to Figure 3GImage materials P1, P2, and P3, after being merged into video template M11, are presented as at least a portion of the image frames in video data P1-V, at least a portion of the image frames in video data P2-V, and at least a portion of the image frames in video data P3-V, respectively, rather than... Figure 2 In the scenario shown, image materials P1, P2, and P3 are presented as static images or simple animations, which provides a better visual effect and enhances the user experience.

[0090] In some embodiments, the mobile phone 10 can also merge video data P1-V, video data P2-V, video data P3-V, video footage V1, video footage V2, and video footage V3 with multiple video templates to obtain multiple editing results for the user to select and preview. This provides the user with more choices and improves the user experience.

[0091] refer to Figure 3H After combining video data P1-V, video data P2-V, video data P3-V, video footage V1, video footage V2, and video footage V3 with video template M11, mobile phone 10 can display the editing result interface U17. The editing result interface U17 can include a preview area U171, an adjustment template option U172, a recombination option U173, and a save option U174. Among them, the preview area U171 is used to display a preview of the editing result; the adjustment template option U172 is used for users to reconfigure the video template; the recombination option U173 is used to trigger the combination of the editing result based on the user-configured video template; and the save option U174 is used to trigger the storage of the editing result on mobile phone 10, sharing with other electronic devices, or uploading to the cloud or home storage device.

[0092] For example, when the system detects that the user has selected video template M12 in the template adjustment option U172 and then clicked the re-composite option U173, the mobile phone 10 can combine video data P1-V, video data P2-V, video data P3-V, video material V1, video material V2, and video material V3 with video template M12 and update the combined editing result in the preview area U171.

[0093] In some embodiments, the mobile phone 10 can also perform video augmentation processing on one or more video materials among the M multimedia materials selected by the user based on user operation, and generate editing results based on the video data after video augmentation processing.

[0094] For example, after detecting that the user clicks the AI ​​video augmentation option U135 on the aforementioned preference selection interface U13, the mobile phone 10 can determine the video material that needs to undergo video augmentation processing from among the M multimedia materials (e.g., the video material that meets the second condition, such as video material V2). Then, the mobile phone 10 can display as follows: Figure 4A The preference selection interface U13″ shown includes options in the media adjustment section for video media requiring video augmentation processing. These options may include AI processing identifiers to indicate to the user that the video media is needed (or will be used, or has already been used) for generating video data. For example, see reference... Figure 4A The options for adjusting the media, specifically for video media V2, may include the AI ​​processing identifier U138.

[0095] In some embodiments, after determining the video material that needs to undergo video augmentation processing, the mobile phone 10 can generate video data corresponding to each video material.

[0096] For example, Figure 4B According to some embodiments of this application, a schematic diagram of video data V2' is shown.

[0097] like Figure 4B As shown, video footage V2 can be a video of someone watering flowers, causing them to grow from seedlings to mature plants. Mobile phone 10 can expand video footage V2 by adding videos of someone watering flowers, allowing them to grow from seeds to seedlings, and videos of flowers growing from mature plants to mature plants, resulting in... Figure 4B The video data shown is V2'.

[0098] In some embodiments, after detecting that the user clicks the generate control U136' on the preference selection interface U13″, the mobile phone 10 can merge video data P1-V, video data P2-V, video data P3-V, video material V1, video data V2', and video material V3 into the first video template.

[0099] For example, Figure 5 According to some embodiments of this application, a schematic diagram is shown of fusing video data P1-V, video data P2-V, video data P3-V, video material V1, video data V2', and video material V3 into a video template M21.

[0100] like Figure 5As shown, the video template M21 has six video slots, such as video slots L13 to L18. The duration of video slot L16 is longer than that of video slot L10. The mobile phone 10 can composite multiple image frames extracted from video material V1 that occupy the same video duration as video slot L13 into video slot L13; multiple image frames extracted from video data P1-V that occupy the same video duration as video slot L14 into video slot L14; and multiple image frames extracted from video data P2-V that occupy the same video duration as video slot L15 into video slot L15. Multiple image frames extracted from video data V2' that occupy the same video duration as the slot duration of video slot L16 are composited into video slot L16. Multiple image frames extracted from video data P3-V that occupy the same video duration as the slot duration of video slot L17 are composited into video slot L17. Multiple image frames extracted from video material V3 that occupy the same video duration as the slot duration of video slot L18 are composited into video slot L18, thus obtaining the editing result.

[0101] Continue to refer to Figure 5 Image materials P1, P2, and P3, after being merged into video template M21, are presented as at least a portion of image frames from video data P1-V, at least a portion of image frames from video data P2-V, and at least a portion of image frames from video data P3-V, respectively. Video material V2 is presented as at least a portion of image frames from video data V2', which has a longer duration. Figure 3G The scenario shown better reflects the content in video material V2, which is beneficial to the user experience.

[0102] It should be noted that, Figures 3A to 5 The process of generating editing results based on user operation by the mobile phone 10 shown is only an example. In other embodiments, the mobile phone 10 may also perform a first processing on at least some of the multimedia materials selected by the user from the M multimedia materials based on other interaction methods, and generate editing results based on the video data after the first processing and the multimedia materials that have not undergone the first processing. This is not limited here.

[0103] Below, in conjunction with Figures 3A to 5 The example shown and Figure 6 This paper introduces the technical solution of this application.

[0104] For example, Figure 6 According to some embodiments of this application, a schematic diagram of the interactive flow of a video generation method is shown. For example... Figure 6 As shown, the method includes:

[0105] S601, the first electronic device detects the user's first operation, wherein the first operation is used to instruct M multimedia materials to be edited into a video. Where M is an integer greater than 1.

[0106] After detecting the user's first operation, the first electronic device can edit M multimedia materials based on S602 to S607.

[0107] For example, the first operation may be to select a control for triggering the editing of the multimedia materials after detecting that the user has selected the above M multimedia materials.

[0108] For example, the first operation could be the user clicking to generate control U136' in the preference selection interface U13', or the user clicking to generate control U144 in the preview interface U14, or the user clicking to generate control U136' in the preference selection interface U13″.

[0109] It should be noted that in some other embodiments, the first operation may be other operations, which are not limited here.

[0110] It should be noted that in other embodiments, the first electronic device may also edit M multimedia materials based on S602 to S607 when it receives a user's voice command, or a first command based on an application or system module in the first electronic device, or a first command sent by another electronic device, and this is not limited here.

[0111] It should be noted that the M multimedia materials may include multimedia materials already stored in the first electronic device, and / or multimedia materials accessible to the first electronic device stored in other electronic devices, and / or multimedia materials captured by the first electronic device.

[0112] S602, the first electronic device determines N multimedia materials from M multimedia materials, wherein the N multimedia materials include R image materials that satisfy the first condition and / or S video materials that satisfy the second condition from the M multimedia materials. Here, N is an integer greater than or equal to 1 and less than or equal to M, R and S are integers greater than or equal to 0 and less than or equal to N, and R+S equals N.

[0113] After detecting the first operation, the first electronic device can identify image and video materials from M multimedia materials. Then, the first electronic device can identify R image materials suitable for image-to-video processing (e.g., image materials that meet the first condition) from the image materials, and / or identify S video materials that need to undergo video augmentation processing (e.g., image materials that meet the second condition) from the video materials.

[0114] For example, in the case where M multimedia materials can include image material P1, image material P2, image material P3, video material V1, video material V2, and video material V3:

[0115] for Figures 3A to 3H In the scenario shown, M multimedia materials can include image material P1, image material P2, image material P3, video material V1, video material V2, and video material V3, and N multimedia materials can include image material P1, image material P2, and image material P3 (as an example of R image materials, R = M = 3); for Figures 4A to 5 In the scenario shown, N multimedia materials can include image material P1, image material P2, image material P3 (as an example of R image materials, where R is 3) and video material V2 (as an example of S video materials, where S is 1, and N = R + S = 4).

[0116] In some embodiments, the first condition may include one or more of the following conditions C11 to C18.

[0117] Condition C11 specifies that the image material is one selected by the user. Based on condition C11, image-to-video processing can be performed on the user-selected image material to meet the user's personalized needs for editing multimedia materials.

[0118] Condition C12 states that at least one subject in the image material is a preset subject. Based on condition C12, image-to-video processing can be performed only on image materials whose subjects include the preset subjects, avoiding the generation of dynamic videos for subjects that are not suitable for video generation (such as static subjects like furniture).

[0119] In some embodiments, the preset subject can be determined based on experience or experimentation. For example, the preset subject may include, but is not limited to: people, plants, landscapes, cities, the sun, the sea, the sky, grasslands, lakes, waterfalls, vehicles, rivers, the sea, sunrise, sunset, starry skies, candles, lanterns, lamps, and night scenes.

[0120] It should be noted that the above-mentioned preset subject is only an example. In other embodiments, the preset subject may include more or fewer types, or split or merge some types. This application embodiment does not limit this. For example, the person type can be divided into portrait, child, adult, smiling back, woman, man, bride, clown, person singing with a microphone, child, Santa Claus, baby, etc. The plant type can be divided into flowers, foliage plants, plum blossom, sunflower, ginkgo, lotus, bouquet, moss, lotus leaf, willow, daisy, carnation, orchid, ivy, rose, tulip, new bud, maple leaf, chrysanthemum, red okra, dandelion, etc.

[0121] Condition C13 specifies that the scene tags in the image material match the preset scene tags. Based on condition C13, image-to-video processing can be performed only on image materials whose scene tags match the preset scene tags, avoiding the generation of dynamic videos for scene types that are not suitable for video generation.

[0122] In some embodiments, preset scene tags may include, but are not limited to, snippets of life, travel, parent-child activities, selfies, scenery, food, cute pets, and happy moments.

[0123] It should be noted that in other embodiments, the preset scene tags may include more or fewer tags, which is not limited here. For example, the preset scene tags may also include sports, driving, dancing, etc.

[0124] Condition C14 states that the image material includes a human figure, and the size of the face in the human figure is greater than or equal to a preset size. Based on condition C14, for image materials containing human figures (denoted as human figure image materials), only human figure image materials with a face size greater than or equal to the preset size can be processed to generate video, thereby avoiding the visual effect of the generated video data being affected by the small size of the face due to the small face size.

[0125] Condition C15 states that the image material includes portraits, and the degree of occlusion of the face in the portrait is less than or equal to the first degree threshold. Based on condition C15, for portrait image material, image-to-video processing can be performed only on portrait image material with a small degree of face occlusion, thereby avoiding incomplete, distorted, or asymmetrical faces in the generated video data due to excessive face occlusion.

[0126] Condition C16 states that the image material includes a human figure, and the degree of occlusion of the human figure is less than or equal to the second degree threshold. Based on condition C16, for human figure image materials, image-to-video processing can be performed only on human figure image materials with a small degree of occlusion, thereby avoiding incomplete or deformed human figures (e.g., missing limbs or incomplete limbs) in the generated video data due to excessive occlusion.

[0127] Condition C17 states that the image material includes portraits, and the clarity of the faces in the portraits is greater than or equal to a clarity threshold. Based on condition C17, for portrait image material, image-to-video processing can be performed only on portrait image material with relatively clear faces, thereby avoiding unclear faces in the generated video data due to unclear faces.

[0128] Condition C18: The image material includes a portrait, and the shooting angle of the portrait is within a preset angle range. Based on condition C18, image-to-video processing is performed on portrait image materials whose shooting angle is within the preset angle range, thereby avoiding poor shooting angles (such as portraits with the subject facing away from the camera or side profile).

[0129] In some embodiments, the shooting angle can refer to the angle of the camera relative to the front of the human body when shooting portrait image material. For example, taking 0° as the front of the human body facing outwards as an example, the preset angle range can be -X° to +Y°, where X and Y can be empirical or experimental values, and X and Y can be the same or different.

[0130] It should be noted that the aforementioned first degree threshold, second degree threshold, clarity threshold, preset angle range, preset size, etc., can be empirical values, experimental values, or values ​​determined by other methods, and are not limited in the embodiments of this application.

[0131] It should be noted that in some other embodiments, the first condition may include more or fewer conditions, which is not limited here.

[0132] In some embodiments, the second condition may include one or more of the following conditions C21 to C24.

[0133] Condition C21 specifies that the video material is the video material selected by the user. Based on condition C21, video augmentation processing can be performed on the user-selected video image material to meet the user's personalized needs for editing multimedia materials.

[0134] Condition C22 states that the video length of the video footage is less than or equal to the preset length. Based on condition C22, short video footage can be extended to prevent this portion of the video from taking up too little time in the edited result and failing to reflect its content.

[0135] It should be noted that the preset duration can be an empirical or experimental value. For example, the preset duration can be 0.5S, 1S, 2S, 3S, 5S or any other arbitrary duration, without any limitation here.

[0136] Condition C23 states that at least one subject in the video footage is a preset subject. Based on condition C23, video augmentation processing can be performed only on video footage whose subjects include the preset subject, avoiding processing subjects unsuitable for video augmentation, which would result in poor quality video data. The preset subject can be referenced from the aforementioned condition C12, and will not be elaborated upon here.

[0137] Condition C24 states that the video quality of the source material is lower than or equal to a video quality threshold. Based on condition C24, video augmentation processing can be performed on video source material with poor quality, thereby preventing the generated edit from also having poor video quality due to the low video quality.

[0138] In some embodiments, the video quality of video footage can be determined based on at least one video parameter of the video footage. For example, video parameters may include, but are not limited to, the sharpness of the subject, the degree of image jitter, and the speed of the subject's movement. Higher subject sharpness corresponds to higher video quality, lower image jitter corresponds to higher video quality, and lower subject movement speed corresponds to higher video quality.

[0139] It should be noted that in other embodiments, the quality of video material can also be determined based on other video parameters, such as resolution, bit depth, color space, video encoding method, etc., which are not limited here.

[0140] It should be noted that in some other embodiments, the second condition may include more or fewer conditions, which is not limited here.

[0141] S603, the first electronic device preprocesses MN multimedia materials and sends N multimedia materials to the second electronic device, wherein the MN multimedia materials are the other multimedia materials other than the N multimedia materials among the M multimedia materials.

[0142] After identifying N multimedia materials, the first electronic device can send the N multimedia materials to the second electronic device.

[0143] For example, for Figures 3A to 3G In the scenario shown, the first electronic device can send image material P1, image material P2, and image material P3 to the second electronic device.

[0144] For example, regarding Figures 4A to 5 In the scenario shown, the first electronic device can send image material P1, image material P2, image material P3 and video material V2 to the second electronic device.

[0145] In some embodiments, a first electronic device can determine the material information of each multimedia material among N multimedia materials and send the determined material information to a second electronic device. For example, the material information of each multimedia material among the N multimedia materials can be used to generate video data corresponding to each multimedia material.

[0146] For example, material information may include, but is not limited to, scene tags (or theme tags), semantic tags, subject tags (used to indicate the main subject in the multimedia material), style tags, material size, material orientation, material type, and video duration (when the material is image material, the video duration can be 0, empty, or other identifiers). Among these, style tags are used to indicate the style of the multimedia material; for example, the style may include, but is not limited to, traditional Chinese style, sports, and cute. Semantic tags are used to describe the content of the multimedia material.

[0147] In some embodiments, the first electronic device can extract multimedia material information based on one or more pre-trained material analysis models. For example, the first electronic device can extract scene tags, semantic tags, style tags, etc., of the multimedia material using one or more classification models. As another example, the first electronic device can determine the subject in the multimedia material using one or more object detection models and use the subject name or type as the subject tag of the multimedia material.

[0148] It should be noted that the material analysis model can be any form of model, including but not limited to convolutional neural networks, recurrent neural networks, attention-based models, graph-based models, multimodal models, support vector machines, mask region-based convolutional neural networks (Mask R-CNN), ghostnet, models based on the "you only look once" (YOLO) structure, and models improved from the above models.

[0149] In some embodiments, the first electronic device may also acquire prompt text input by the user, which instructs the generation of video data for each multimedia material (e.g., a first prompt text corresponding to each image material and a second prompt text corresponding to each video material). The prompt text corresponding to each multimedia material can serve as a prompt message to instruct the image-to-video model / video augmentation model to generate video data based on the content of the prompt text.

[0150] It should be noted that each of the N multimedia materials can correspond to one prompt text, or multiple multimedia materials can correspond to one prompt text, or some of the N multimedia materials can have corresponding prompt texts while others do not.

[0151] In some embodiments, a first electronic device may determine the material information of M multimedia materials and send the material information of the M multimedia materials to a second electronic device. For example, the material information of the M multimedia materials may be used to determine one or more video templates that match the M multimedia materials from a plurality of video templates.

[0152] In some embodiments, preprocessing may include, but is not limited to, determining the highlight information of video footage, determining the composition information of image footage, and performing processing on image footage such as sharpening, adjusting contrast, and increasing clarity.

[0153] It should be noted that highlight information includes the location and duration of highlight image frames (or highlight segments) in video footage. A highlight image frame or highlight segment can refer to one or more image frames in the video footage that are representative, the most exciting, or that best reflect the content or are the most crucial. A video clip may include one or more highlight image frames (or highlight segments).

[0154] In some embodiments, the first electronic device can extract highlight information from video footage using a pre-trained artificial intelligence model or algorithm.

[0155] It should be noted that the composition information may include information used for cropping, scaling, rotating, and other processing of the image material, so as to determine the image area and blending angle that are suitable for merging into the video template.

[0156] It should be noted that in other embodiments, preprocessing may include more or fewer processing methods, which is not limited here.

[0157] It should be noted that S603 can also be divided into multiple steps, such as preprocessing MN multimedia materials and sending N multimedia materials to the second electronic device. The preprocessing of MN multimedia materials and the sending of N multimedia materials to the second electronic device can be performed in parallel or sequentially. This application embodiment does not limit this.

[0158] S604, the second electronic device performs a first process on N multimedia materials to obtain N video data, wherein the N video data includes R video data obtained by performing image-to-video processing on R image materials and / or S video data obtained by performing video augmentation processing on S video materials.

[0159] After receiving N multimedia materials, the second electronic device can perform a first processing on each of the N multimedia materials to obtain N video data.

[0160] For example, given R image clips out of N multimedia materials, the second electronic device can input each image clip and its corresponding first prompt into the image-generated video model to obtain the video data for each image clip. See below for details. Figure 8 The embodiments shown are not described in detail here.

[0161] It should be noted that the first prompt information for each of the R image materials may include a first prompt text sent by the first electronic device, and / or a second prompt text generated by the second electronic device based on the material information of each image material and / or each image material. For example, the second electronic device can input the material information of the image material and / or each image material into the prompt word enhancement model to obtain the second prompt text corresponding to that image material.

[0162] It should be noted that the prompt enhancement model can be any model that can obtain prompt information related to the content and style of the multimedia material based on the material information and / or the multimedia material itself. This application embodiment does not limit this. For example, the prompt enhancement model can include, but is not limited to, multimodal models, large language models, OpenFlamingo, Kosmos-2, ReAct, Reflexion, and models improved based on the above models.

[0163] For example, for Figures 3A to 5 In the scenario shown, the R image materials can include image materials P1 to P3. The second prompt text for image material P1 can be "a person, the person's arms go from hanging down naturally to crossing in front of the body and then hugging the chest". The second prompt text for image material P2 can be "tree branches and leaves, the tree branches and leaves sway in the wind". The prompt text for image material P3 can be "clouds and trees, the clouds sway in the wind".

[0164] In some embodiments, the second prompt text may further include the duration of the video data obtained by image-to-video processing of the image material (denoted as the image-to-video duration). For example, for image material P3, the second prompt text could be "Clouds and trees, clouds drifting in the wind, total duration 3 seconds," where 3 seconds is the image-to-video duration of image material P3. In this case, the image-to-video model can generate video data P3-V with a duration of 3 seconds based on image material P3.

[0165] In some embodiments, for any first image material among N multimedia materials, the second prompt text corresponding to the first image material can be obtained by inputting one or more of the main label, scene label, and semantic label of the first image material (and / or M multimedia materials) into the prompt word enhancement model.

[0166] It should be noted that in some other embodiments, the first prompt message may also include other content, which is not limited here.

[0167] For example, given S video clips out of N multimedia materials, the second electronic device can input each video clip and its corresponding second prompt into the video augmentation model to obtain the video data for each video clip. See below for details. Figure 9 The embodiments shown are not described in detail here.

[0168] It should be noted that the second prompt information for each of the S video clips may include a third prompt text sent by the first electronic device, and / or a fourth prompt text generated by the second electronic device based on the clip information and / or the clips themselves. For example, the second electronic device can input the clip information and / or the clips themselves into the prompt enhancement model to obtain the fourth prompt text corresponding to that video clip.

[0169] In some embodiments, for any first video material among N multimedia materials, the fourth prompt text corresponding to the first video material can be obtained by inputting one or more of the subject label, scene label, and semantic label of the first video material or (and / or M multimedia materials) into the prompt word enhancement model.

[0170] It should be noted that the prompt enhancement model, image-generated video model, and video augmentation model can run on a second electronic device or other electronic devices, which is limited here.

[0171] For example, corresponding Figures 4A to 5 In this case, the S video materials can include video material V2, and the fourth prompt text of video material V2 can be "flowers, expanding the flowers from seed to seedling before the video, and expanding the flowers in bloom after the video".

[0172] In some embodiments, the fourth prompt text may also include the video duration (denoted as the video augmentation duration) obtained by augmenting the video data. For example, the fourth prompt text could be "Flowers, augmenting the video from seed to seedling before the video, and augmenting the video after the video to show the flowers blooming, augmenting it into a 10-second video," where 10 seconds is the video augmentation duration of video material V2. In this case, the video augmentation model can augment video material V2 into video data V2' with a video duration of 10 seconds.

[0173] It should be noted that in some other embodiments, the second prompt message may also include other content, which is not limited here.

[0174] S605, the second electronic device sends N video data to the first electronic device.

[0175] After obtaining N video data points, the second electronic device can send N video data points to the first electronic device.

[0176] For example, the second electronic device can send N video data to the first electronic device itself, or it can compress and encode the N video data separately and then send them to the first electronic device. Alternatively, it can send the download address (e.g., a Uniform Resource Locator (URL) address) of the N video data to the first electronic device.

[0177] For example, for Figures 3A to 3E In the scenario shown, the N video data can include video data P1-V, video data P2-V, and video data P3-V, or the URLs corresponding to video data P1-V, video data P2-V, and video data P3-V.

[0178] For example, for Figures 4A to 5 In the scenario shown, the N video data may include video data P1-V, video data P2-V, video data P3-V, and video data V2', or the URLs corresponding to video data P1-V, video data P2-V, video data P3-V, and video data V2'.

[0179] S606, the first electronic device combines N video data and MN multimedia materials into a first video template to obtain the first video.

[0180] After acquiring N video data points, such as receiving N video data points from a second electronic device or downloading N video data points based on a download address sent by the second electronic device, the first electronic device can combine the N video data points and MN multimedia materials into a first video template to obtain the first video.

[0181] For example, given N video data and MN multimedia materials containing image materials, the first electronic device can integrate the image materials into image slots in the first video template and generate multiple frames of images with the slot duration corresponding to the material slot based on the presentation format defined by the image slot.

[0182] For example, given N video data points and MN multimedia materials containing image elements, the first electronic device can merge the video elements into the video slots of the first video template. If the duration of the video slot is the same as the duration of the video material, the first electronic device can directly merge the video material into the slot. If the duration of the video slot is less than the duration of the video material, the first electronic device can extract multiple image frames (or video clips) from the video material whose duration matches the duration of the video slot, and then merge these extracted image frames (or video clips) into the video slot.

[0183] For example, for Figure 3G As shown, the first electronic device can merge video data P1-V, video data P2-V, and video data P3-V (as examples of N video data) into video slots L8, L9, and L11 of video template M11 (as an example of the first video template), and merge video material V1, video material V2, and video material V3 (as examples of MN multimedia materials) into video slots L7, L10, and L12, respectively, to obtain the first video.

[0184] For example, for Figure 5 As shown, the first electronic device can merge video data P1-V, video data P2-V, video data P3-V, and video data V2' (as an example of N video data) into video slots L14 to L17 in video template M21 (as an example of the first video template), and merge video material V1 and video material V3 (as an example of MN multimedia materials) into video slots L7 and L12, respectively, to obtain the first video.

[0185] It should be noted that the number of media slots in the first video template can be greater than or equal to M. When the number of media slots in the first video template is greater than M, when merging N video data points and MN multimedia media points into the first video template, the N video data points and MN multimedia media points can be merged into the M media slots of the first video template respectively, and the media slots in the first video template that have not been merged with multimedia media points should be deleted to ensure the continuity of the resulting edit.

[0186] For example, when M=2, the first video template includes three consecutive media slots (denoted as media slot A1, media slot A2, and media slot A3). The video time intervals corresponding to media slots A1, A2, and A3 are from time t1 to time t2, from time t2 to time t3, and from time t3 to time t4, respectively. If media slots A1 and A3 contain multimedia content, but media slot A2 does not contain multimedia content, the template content from time t2 to time t3 can be deleted. This ensures that the editing result includes video content with a duration of t2-t1 obtained by merging one multimedia content into media slot A1, and video content with a duration of t3-t4 obtained by merging another multimedia content into media slot A2.

[0187] It should be noted that the first video template can be a video template that matches M multimedia materials from among multiple video templates.

[0188] In some embodiments, the first video template may be a video template that matches the M multimedia materials, determined by the first electronic device from multiple video templates based on the material information of the M multimedia materials.

[0189] In some embodiments, the first video template may also be a video template that matches the M multimedia materials, which is determined by the second electronic device (or other electronic device) from multiple video templates based on the material information of the M multimedia materials and sent to the first electronic device.

[0190] In some embodiments, the first video template may also be a video template selected by the user from multiple video templates.

[0191] In some embodiments, the first video template can be a video template whose at least one template parameter matches N video data and MN multimedia materials. The template parameters include at least one of the following: template size, template orientation, template semantic tags, template scene tags, number of material slots, slot duration of material slots, and slot type of material slots. The template semantic tags are used to indicate the multimedia materials to which the video template is applicable; the template scene tags are used to indicate the scene or theme to which the video template belongs.

[0192] For example, the first video template can be a video template that meets the third condition from among a plurality of video templates. The third condition may include one or more of the following conditions C31 to C36.

[0193] Condition C31 states that the scene tags of the first video template match the scene tags of the M multimedia materials. Based on condition C31, it can be ensured that the applicable scene or theme of the first video template matches the scene and theme of the M multimedia materials.

[0194] Condition C32 states that the number of media slots T in the first video template is greater than or equal to M. Based on condition C32, it can be ensured that the first video template has enough media slots to accommodate N video data and MN multimedia media, avoiding the need for user-selected multimedia media.

[0195] Condition C33: The template orientation of the first video template matches the orientation of M multimedia materials. The template orientation can be horizontal or vertical, and the material orientation can also be horizontal or vertical. A horizontal template orientation indicates that the width of the video in the resulting edit is greater than or equal to its height; a vertical template orientation indicates that the width of the video in the resulting edit is less than or equal to its height. Similarly, a horizontal orientation for multimedia materials indicates that the width of the multimedia material is greater than or equal to its height; a vertical orientation for multimedia materials indicates that the width of the multimedia material is less than or equal to its height.

[0196] In some embodiments, matching the template direction of the first video template with the material direction of the M multimedia materials can be achieved by having the template direction of the first video template be the same as the material direction of most of the M multimedia materials.

[0197] Based on condition C33, it can prevent some multimedia materials from being stretched, cropped, or filled with black / white or other content due to the mismatch between the template orientation of the first video template and the material orientation of the M multimedia materials.

[0198] Condition C34 states that the longest duration of any media slot in the first video template is less than or equal to the total video duration of all video media in the M multimedia media slots. Based on condition C34, this prevents the video media in the M multimedia media from failing to fill the corresponding video duration of the media slot due to an excessively long slot duration.

[0199] Condition C35 states that the number of video slots in the first video template whose slot duration is less than the first duration is greater than or equal to N. The first duration is the video duration of each of the N video data. In this implementation, each of the N video data has the same duration. Based on condition C35, it can be ensured that the material slots for merging the N video data can be merged into the image frames of the N video data, avoiding the appearance of solid color (such as white or black) or image frames filled with other content in the editing result due to excessively long slot durations.

[0200] In one embodiment, condition C35 can also be replaced with other equivalent conditions. For example, the proportion of video slots in the first video template whose slot duration is less than the first duration is greater than or equal to N / M, etc.

[0201] Condition C36 states that the distribution of slot types in the media slots of the first video template matches the media types of the N video data and MN multimedia materials. For example, the number of video slots in the first video template can be greater than or equal to the number of video materials in the N video data and MN multimedia materials, and the number of image slots in the first video template can be greater than or equal to the number of image materials in the MN multimedia materials. Based on condition C36, it can be ensured that the N video data and MN multimedia materials can be merged into media slots with the same slot type as the multimedia materials.

[0202] It should be noted that in other embodiments, the third condition may include more or fewer conditions, which is not limited here.

[0203] S607, the first electronic device displays the first video.

[0204] After obtaining the first video, the first electronic device can display the first video.

[0205] For example, for Figure 3H As shown, the first electronic device can display the first video in the preview area U171 of the editing result interface U17.

[0206] based on Figure 6 The method shown can ensure that the image materials among the N multimedia materials are presented in the form of dynamic video in the first video, thereby improving the visual effect of the first video. And / or, it can ensure that the video materials among the N multimedia materials can occupy a longer duration in the first video, thereby ensuring that the content of the video materials can be accurately presented in the first video.

[0207] The following describes a technical solution for determining the aforementioned R image materials suitable for image-to-video processing.

[0208] For example, Figure 7 According to some embodiments of this application, a schematic flowchart for determining image material suitable for performing image-to-video processing is shown. The execution entity of this process is a first electronic device. Figure 7 As shown, the method includes:

[0209] S701, determine the image material among M multimedia materials.

[0210] After acquiring M multimedia materials, the first electronic device can identify the image materials among the M multimedia materials.

[0211] For example, in some embodiments, the first electronic device may determine the image material in the M multimedia materials based on the file type or file extension of the M multimedia materials.

[0212] For example, in a gallery application, the Uniform Resource Identifier (URI) of multimedia assets can be recorded. A first electronic device can then determine the file type or file extension of M multimedia assets based on their URIs. The first electronic device can then identify the image assets within the M multimedia assets based on their file type or file extension.

[0213] For example, file extensions for image materials can include, but are not limited to, ".jpg", ".jpeg", ".png", ".gif", ".bmp", ".tiff", ".psd", ".heic", ".raw", ".apng", ".avif", ".mng", ".svg", ".ai", ".eps", ".cdr", ".webp", etc.

[0214] S702, determine the scene tags for each image material.

[0215] After identifying the image elements from the M multimedia materials, the first electronic device can determine the scene label for each image element. For example, the first electronic device can determine the scene label for each image element based on the aforementioned material analysis model.

[0216] In some embodiments, scene tags may include, but are not limited to, snippets of life, travel, parent-child activities, selfies, scenery, food, cute pets, happy moments, still life (such as furniture), sports, dance, driving, etc.

[0217] S703, determine one or more image materials whose scene label is a preset scene label, and determine the subject in the one or more image materials.

[0218] After obtaining the scene tags of each image material, if the scene tag of an image is a preset scene tag, then the scene image may be suitable for performing image-to-video processing (equivalent to satisfying the aforementioned condition C13). The first electronic device can use the image material as an image material among the aforementioned one or more image materials, and determine the subject of the one or more image materials.

[0219] In some embodiments, the electronic device may determine the subject in each of the one or more image materials based on an object detection model, a classification model, or other models. For example, for the aforementioned image material P1, the subject may be a human figure; for the aforementioned image material P2, the subject may be a tree; and for the aforementioned image material P3, the subject may be both a tree and clouds.

[0220] S704, among one or more image materials, image materials whose main subject includes a human image and whose human image features meet the human image feature conditions are identified as image materials suitable for performing image-to-video processing.

[0221] In some embodiments, portrait features may include the size of the face, the degree of occlusion of the face, the degree of occlusion of the portrait, the clarity of the face, and the shooting angle corresponding to the portrait. Portrait feature conditions may include at least one of the following conditions: the size of the face is greater than or equal to a preset size; the degree of occlusion of the face is less than or equal to a first degree threshold; the degree of occlusion of the portrait is less than or equal to a second degree threshold; the clarity of the face is greater than or equal to a clarity threshold; and the shooting angle corresponding to the portrait is within a preset angle range. That is, portrait feature conditions may include one or more of the aforementioned conditions C14 to C18.

[0222] S705, determine one or more image materials whose main subject includes a preset subject but does not include human figures as image materials suitable for performing image-to-video processing.

[0223] Based on S701 to S705, the R image materials can include image materials from M multimedia materials whose scene label is a preset scene label, whose subject includes a human image, and whose human image features meet the human image feature conditions, as well as image materials from M multimedia materials whose scene label is a preset scene label, whose subject does not include a human image, and whose subject is a preset subject.

[0224] S706, Determine the style tags for image materials suitable for performing image-to-video processing.

[0225] In some embodiments, after determining R image clips suitable for image-to-video processing, the first electronic device can determine style tags corresponding to each of the R image clips. The style tags of the image clips can be used to generate prompts corresponding to the image clips, which can instruct the image-to-video model to generate video data based on the image clips.

[0226] It should be noted that S706 is an optional step, meaning that the first electronic device does not need to determine the style tags of the R image materials.

[0227] based on Figure 7 The method shown allows the first or second electronic device to perform image-to-video processing only on image materials suitable for image-to-video processing, thereby determining the quality of the obtained video data and avoiding the waste of computing resources caused by performing image-to-video processing on image materials that are not suitable for image-to-video processing.

[0228] The following describes a technical solution for generating R video data based on R image materials.

[0229] For example, Figure 8 According to some embodiments of this application, a schematic diagram of a process for generating R video data based on R image materials is shown. The execution entity of this process is a second electronic device. Figure 8 As shown, the method includes:

[0230] S801, obtain the style tags and / or subject tags of R image assets.

[0231] The second electronic device can acquire style tags and / or subject tags for R image materials.

[0232] For example, the second electronic device can receive style tags and / or subject tags of R image materials from the first electronic device, or it can determine the style tags and / or subject tags of R image materials through a material analysis model.

[0233] S802, Generate a second cue text for each image material based on the style tag and / or subject tag of each image material in the R image materials.

[0234] After obtaining the style tags and / or subject tags of R image materials, the second electronic device can input the style tags and / or subject tags of each image material into the prompt enhancement model to obtain the second prompt text corresponding to each image material.

[0235] For example, for the aforementioned image material P2, the main label can be "tree". The second electronic device can input "tree" into the prompt word enhancement model to obtain the second prompt text "branches and leaves, branches and leaves swaying in the wind".

[0236] S803, based on each image material and the second prompt text of each image material, calls the image-generated video model to obtain R video data corresponding to each of the R image materials.

[0237] After obtaining the second prompt text for each image material, for any given image material, the second electronic device can use the second prompt text of that image material as the first prompt information for that image material, and input the first prompt information and the image material into the image-generated video model to obtain the video data corresponding to that image material.

[0238] In some embodiments, if the second electronic device receives a first prompt text for an image material, the second electronic device uses the first prompt text and the second prompt text of the image material as the first prompt information for the image material, and inputs the first prompt information and the image material into the image-generated video model to obtain the video data corresponding to the image material.

[0239] In some embodiments, the first prompt information may further include the image-generated video duration of each of the R image materials.

[0240] For example, for the aforementioned image material P2, the main label could be "tree." The second electronic device could input "tree" into the prompt enhancement model to obtain the second prompt text "branches and leaves, branches and leaves swaying in the wind." The first prompt information corresponding to image material P2 could be "branches and leaves, branches and leaves swaying in the wind, video duration 3 seconds," where 3 seconds is the duration of the image-generated video of image material P2. The image-generated video model could generate video data with a duration of 3 seconds based on image material P2 and the first prompt information "branches and leaves, branches and leaves swaying in the wind, video duration 3 seconds."

[0241] In some embodiments, the duration of the generated video of the R image materials can be the same (e.g., the first duration mentioned above) or different, without limitation.

[0242] In some embodiments, the duration of the generated video for each image material can also be determined based on the subject, scene, etc., of each image material. For example, image materials with different subjects or different scenes may correspond to the same or different generated video durations. Exemplarily, an electronic device (a first electronic device or a second electronic device) can determine the duration of the generated video for each image material based on a first correspondence relationship. The first correspondence relationship includes a one-to-one correspondence between combinations of subjects and scenes and the generated video duration. For any combination of subjects and scenes, the generated video duration corresponding to that combination can be included in the first correspondence relationship. For any image material, the electronic device can determine the generated video duration corresponding to the subject and scene of that image material in the first correspondence relationship as the generated video duration of that image material.

[0243] In some embodiments, the first duration may also be determined based on the video clips in the M multimedia materials. For example, the first duration may be greater than or equal to the average, maximum, or minimum duration of the highlight image frames (or highlight segments) of each video clip in the M multimedia materials. As another example, the first duration may be greater than or equal to the average, maximum, or minimum video duration of each video clip in the M multimedia materials.

[0244] It should be noted that in other embodiments, the duration of the generated video for each image material can also be determined based on other methods, which are not limited here.

[0245] based on Figure 8 As shown, the second electronic device can call the image-generated video model to generate video data corresponding to the image material based on the style tags and / or subject tags of the image material, so that the generated video data can maintain the same or similar style as the original image material.

[0246] The following describes a technical solution for generating S video data based on S video materials.

[0247] For example, Figure 9 According to some embodiments of this application, a schematic diagram of a process for generating S video data based on S video clips is shown. The execution entity of this process is a second electronic device. Figure 9 As shown, the method includes:

[0248] S901, retrieve the style tags and / or subject tags for S video clips.

[0249] The second electronic device can acquire style tags and / or subject tags for S video clips.

[0250] For example, the second electronic device can receive style tags and / or subject tags of S video clips from the first electronic device, or it can determine the style tags and / or subject tags of the S video clips through a material analysis model.

[0251] S902, generate a fourth cue text for each video clip based on the style tags and / or subject tags and / or video clips of each of the S video clips.

[0252] After obtaining the style tags and / or subject tags of S video clips, the second electronic device can input the style tags and / or subject tags of each video clip into the prompt enhancement model to obtain the fourth prompt text corresponding to each video clip.

[0253] For example, for the aforementioned video material V2, the main tag can be "flowers" and the style tag can be "plant growth". The second electronic device can input "flowers", "plant growth" and video material V2 into the prompt word enhancement model to obtain the fourth prompt text corresponding to video material V2: "flowers, expand the flowers from seed to seedling before the video, and expand the flowers blooming after the video".

[0254] S903, based on each video material and the fourth prompt text of each video material, calls the video augmentation model to obtain S video data corresponding to each of the S video materials.

[0255] After obtaining the fourth prompt text of each video material, for any given video material, the second electronic device can use the fourth prompt text of the video material as the second prompt information of that video material, and input the second prompt information and the video material into the video augmentation model to obtain the video data corresponding to that video material.

[0256] In some embodiments, if the second electronic device receives a third prompt text for a video clip, the second electronic device uses the third and fourth prompt texts of the video clip as the second prompt information for the video clip, and inputs the second prompt information and the video clip into the video augmentation model to obtain the video data corresponding to the video clip.

[0257] In some embodiments, the second prompt information may also include the extended video duration of each video element.

[0258] In some embodiments, the extended video duration of the S video materials can be the same duration (e.g., the first duration) or different durations, without limitation.

[0259] In some embodiments, the video extension duration for each video clip can also be determined based on the subject, scene, etc., of each video clip. For example, video clips with different subjects or different scenes can correspond to the same or different video extension durations. Exemplarily, an electronic device (a first electronic device or a second electronic device) can determine the video extension duration for each video clip based on a second correspondence relationship. The second correspondence relationship includes a one-to-one correspondence between combinations of subjects and scenes and video extension durations. For any combination of subjects and scenes, the second correspondence relationship can include the video extension duration corresponding to that combination. For any video clip, the electronic device can determine the video extension duration corresponding to the subject and scene of that video clip in the second correspondence relationship as the video extension duration of that video clip.

[0260] It should be noted that in other embodiments, the video extension duration of each video material can also be determined in other ways, which are not limited here.

[0261] based on Figure 9 As shown, the second electronic device can call the video augmentation model to generate video data corresponding to the video material based on the style tags and / or subject tags of the video material, so that the generated video data can maintain the same or similar style as the original video material.

[0262] The following describes a technical solution for determining the first video template that matches M multimedia materials among multiple video templates.

[0263] For example, Figure 10 According to some embodiments of this application, a flowchart illustrating a process for determining a first video template that matches M multimedia materials from a plurality of video templates is shown. The execution entity of this process can be a first electronic device, a second electronic device, or other electronic devices. The following description uses the example of a second electronic device determining a first video template that matches M multimedia materials from a plurality of video templates. Figure 10As shown, the method includes:

[0264] S1001, Obtain material information for M multimedia materials. The material information may include one or more of the following: scene tags, semantic tags, material size, material orientation, material type, and video duration.

[0265] The second electronic device can acquire material information for M multimedia materials.

[0266] For example, the second electronic device can receive material information for M multimedia materials from the first electronic device. Alternatively, the second electronic device can also receive M multimedia materials from the first electronic device and run a material analysis model to generate material information for each multimedia material.

[0267] S1002, determine at least one video template among multiple video templates whose scene tag, template size, template orientation, number of material slots, and slot type match the material information of M multimedia materials.

[0268] After obtaining the material information of M multimedia materials, the first electronic device can determine at least one video template among multiple video templates whose scene tag, template size, template orientation, number of material slots, and type of material slots match the material information of the M multimedia materials.

[0269] For example, if the scene tags of a video template match the scene tags of M multimedia materials, the template size (e.g., resolution) matches the size of the M multimedia materials, the template orientation matches the orientation of the materials, the number of material slots is greater than or equal to M, and the slot type of the material slots matches the material information of the M multimedia materials, then the video template can be a template among the at least one video template. Specifically, if the number of video slots in a video template is greater than or equal to the number of video data and the number of video materials in the M multimedia materials, and the number of image slots in the video template is greater than or equal to the number of image materials in the M multimedia materials, then the slot type of the video template's material slots matches the material information of the M multimedia materials.

[0270] S1003, determine at least one candidate video template in which the slot duration of at least N material slots in at least one video template matches the video duration of N video data.

[0271] After determining at least one video template, the second electronic device can select at least one candidate video template from the at least one video template. In each candidate video template, the slot duration of at least N material slots matches the video duration of N video data.

[0272] For example, in some embodiments, a video template whose slot duration of at least N material slots matches the video duration of N video data can refer to a video model that includes at least N video slots, where the slot duration of the i-th video slot among the N video slots is less than or equal to the video duration of the i-th video data among the N video data, where i is an integer greater than or equal to 1 and less than or equal to N. For example, when the video duration of all N video data is a first duration, a video template whose slot duration of at least N video slots matches the video duration of the N video data can be: a video template that includes at least N video slots whose slot duration is less than or equal to the first duration.

[0273] S1004, the candidate video template with the highest matching degree between the template semantic tags and the semantic tags of M multimedia materials is determined as the first video template.

[0274] After obtaining the candidate video templates, the second electronic device can determine the matching degree between the semantic tags of the M multimedia materials and the semantic tags of each candidate video template, and determine the candidate video template with the highest matching degree between the template semantic tags and the semantic tags of the M multimedia materials as the first video template.

[0275] In some embodiments, the semantic tags of the M multimedia materials and the semantic tag of the candidate video template can be the similarity between the semantic tags of the M multimedia materials and the semantic tags of the candidate video template. Based on the first video template determined in S1004, it can be ensured that the first video template is more suitable for the M multimedia materials, which is beneficial to improving the visual effect of the obtained first video.

[0276] It should be noted that, Figure 10 The method of determining the first video template shown is only one example. In other embodiments, the first electronic device or the second electronic device may also determine the first video template from multiple video templates in other ways, which is not limited here.

[0277] This application also provides a video generation method applied to a first electronic device. For example, Figure 11 According to some embodiments of this application, a flowchart of a video generation method is shown. The execution entity of this process can be a first electronic device, such as... Figure 11 As shown, the method includes:

[0278] S1101, Receive a first instruction, wherein the first instruction instructs to edit M multimedia materials into a video, the M multimedia materials include at least one image material and / or at least one video material, and M is an integer greater than 1.

[0279] In some embodiments, the first electronic device can be a terminal device, which can receive a first instruction upon detecting a user's first operation. For details, please refer to the aforementioned S601, which will not be repeated here.

[0280] For example, for Figures 3A to 5 In the scenario shown, the first operation may include the operation of clicking to generate control U136' in the user preference selection interface U13' detected by mobile phone 10 (as an example of the first electronic device), or the operation of the user clicking to generate control U144 in the preview interface U14, or the operation of the user clicking to generate control U136 in the preference selection interface U13″. The M multimedia materials may include image material P1, image material P2, image material P3, video material V1, video material V2, and video material V3.

[0281] In some embodiments, the first electronic device may also be a cloud device (e.g., a server or other computing device), and the first instruction may be sent to the first electronic device by another electronic device.

[0282] The first instruction can also be an instruction triggered by the first electronic device based on its own operating logic, or an instruction triggered by an application or system module in the first electronic device.

[0283] It should be noted that in some other embodiments, the first electronic device may also receive the first instruction under other circumstances. The embodiments of this application do not limit the way the user receives the first instruction.

[0284] S1102, obtain N video data, where the N video data are obtained by performing a first processing on N multimedia materials out of M multimedia materials respectively, and N is an integer greater than or equal to 1 and less than or equal to M.

[0285] Upon receiving the first instruction, the first electronic device can first determine N multimedia materials out of M multimedia materials. For example, the N multimedia materials may include R image materials suitable for performing image-to-video processing, and / or S video materials from the M multimedia materials that need to undergo video augmentation processing. Here, R and S are integers greater than or equal to 1 and less than or equal to N, and R+S equals N.

[0286] For example, the N multimedia materials may include image materials that satisfy the first condition from the M multimedia materials, and / or video materials that satisfy the second condition from the M multimedia materials. The method by which the first electronic device determines the N multimedia materials can be referred to the aforementioned S602, and will not be repeated here. For example, for Figures 3A to 3HIn the scenario shown, N multimedia materials can include image material P1, image material P2, and image material P3 (as an example of R image materials, R = M = 3); for Figures 4A to 5 In the scenario shown, N multimedia materials can include image material P1, image material P2, image material P3 (as an example of R image materials, where R is 3) and video material V2 (as an example of S video materials, where S is 1, and N = R + S = 4).

[0287] After determining N multimedia materials, for any image material among the N multimedia materials, the first electronic device or the second electronic device can perform image-to-video processing on the image material to obtain the corresponding video data; for any video material among the N multimedia materials, the first electronic device or the second electronic device can perform video augmentation processing on the video material to obtain the corresponding video data. For example, if the N multimedia materials include a first image material, image-to-video processing can be performed on the first image material (as an example of the first processing) to obtain first video data. As another example, if the N multimedia materials include a first video material, video augmentation processing can be performed on the first video data (as an example of the first processing) to obtain second video data, wherein the video duration of the second video data is longer than the video duration of the first video material. For details, please refer to the aforementioned S604, S801 to S803, and S901 to S904, which will not be elaborated upon here.

[0288] For example, for Figures 3A to 3H In the scenario shown, the first image material can be any one of image material P1, image material P2, or image material P3, and the first video data can be video data P1-V corresponding to image material P1, or video data P2-V corresponding to image material P2, or video data P3-V corresponding to image material P3.

[0289] For example, for Figures 4A to 5 In the scenario shown, the first image material can be any one of image material P1, image material P2, or image material P3. The first video data can be video data P1-V corresponding to image material P1, or video data P2-V corresponding to image material P2, or video data P3-V corresponding to image material P3. The first video material can be video material V2, and the second video data can be video data V2'.

[0290] In some embodiments, the first video data is generated by the first electronic device calling a first model (e.g., the aforementioned image-generated video model) based on the first image material, or the first video data is received by the first electronic device after sending the first image material to the second electronic device; and / or, the second video data is generated by the first electronic device calling a second model (e.g., the aforementioned video augmentation model) based on the first video material, or the second video data is received by the first electronic device after sending the first video material to the second electronic device.

[0291] In some embodiments, the video duration of the first video data is determined based on the video duration of each video material in at least one video material, or the video duration of the first video data is a preset duration (e.g., a first duration). For details regarding the image-generated video duration of image materials and / or the video extension duration of video materials in S604, S803, and S804, please refer to these documents; they will not be elaborated upon here.

[0292] In some embodiments, the first video data is generated by a first electronic device or a second electronic device by calling a first model (such as the aforementioned image-generated video model) based on the first image material and the first prompt information. For example, the first electronic device can run the first model to generate the first video data based on the first image material. Alternatively, the first electronic device can send the first image material to the second electronic device, which then runs the first model to generate the first video data and sends the first video data back to the first electronic device. For details, please refer to the aforementioned S604 and S803, which will not be elaborated upon here.

[0293] In some embodiments, the second video data is generated by the first electronic device or the second electronic device calling a second model (such as the aforementioned video augmentation model) based on the first video material and the second prompt information. For example, the first electronic device can run the second model to generate the second video data based on the first video material. Alternatively, the first electronic device can send the first video material to the second electronic device, and the second electronic device can run the second model to generate the second video data based on the second video material, and then send the second video data back to the first electronic device. For details, please refer to the aforementioned S604 and S903, which will not be elaborated upon here.

[0294] S1103, Generate a first video based on N video data and MN multimedia materials from M multimedia materials, wherein MN multimedia materials are multimedia materials other than N multimedia materials from the M multimedia materials.

[0295] After obtaining N video data points, the first electronic device can use a first video template to generate a first video based on the N video data points and MN multimedia materials. For example, the first electronic device can merge the N video data points and MN multimedia materials into the first video template to generate the first video. For example, the first video template can include T (T is greater than or equal to M) material slots, and the first electronic device can obtain the first video by merging the N video data points and MN multimedia materials into M material slots out of the T material slots.

[0296] In some embodiments, the first video template is a video template whose template parameters match at least one of a plurality of video templates with N video data and MN multimedia materials, or a video template selected by the user from a plurality of video templates, wherein the template parameters include at least one of template size, template orientation, template semantic tags, template scene tags, number of material slots, slot duration of material slots, and slot type of material slots.

[0297] In some embodiments, the first video template satisfies a third condition, wherein the third condition includes at least one of the following conditions: the template scene tag of the first video template matches the scene tags of M multimedia materials; the number of material slots in the first video template is greater than or equal to M; the template direction of the first video template matches the template direction of the M multimedia materials; the slot length of the first material slot with the longest slot length in the material slots of the first video template is less than or equal to the video length of each video material in at least one video material; the number of material slots in the material slots of the first video template whose slot length is less than a first duration is greater than or equal to N, where the first duration is the video length of each video data in the N video data; and the distribution of the slot types of the material slots of the first video template matches the material types of the N video data and the M multimedia materials.

[0298] In some embodiments, the first video template is the video template whose semantic tags have the highest similarity to the semantic tags of M multimedia materials among one or more video templates that satisfy the third condition. For example, after determining one or more video templates that satisfy the third condition, the first electronic device (or the second electronic device, or the third electronic device) can determine the similarity between the semantic tags of each of the one or more video templates and the semantic tags of the M multimedia materials. Then, the first electronic device (or the second electronic device, or the third electronic device) can determine the video template whose semantic tags have the highest similarity to the semantic tags of the M multimedia materials as the first video template.

[0299] In some embodiments, the first video template is received by the first electronic device from other electronic devices (e.g., a second electronic device, a third electronic device, etc.), or the first electronic device determines it from a plurality of video templates.

[0300] It should be noted that the method for determining the first video template from multiple video templates can refer to the aforementioned S606, S1001 to S1004, and will not be repeated here.

[0301] based on Figure 11 The method shown can ensure that the image materials among the N multimedia materials are presented in the form of dynamic video in the first video, thereby improving the visual effect of the first video. And / or, it can ensure that the video materials among the N multimedia materials can occupy a longer duration in the first video, thereby ensuring that the content of the video materials can be accurately presented in the first video.

[0302] This application also provides a video generation apparatus for implementing the video generation methods provided in the foregoing embodiments.

[0303] For example, Figure 12 According to some embodiments of this application, a schematic diagram of the structure of a video generation apparatus 100 is shown.

[0304] like Figure 12 As shown, the video generation device 100 includes a material analysis module 101, a template matching module 102, an image-generated video module 103, a video expansion module 104, a preprocessing module 105, and a compositing module 106.

[0305] The material analysis module 101 is used to analyze M multimedia materials, determine the material information of each material, the portrait features in the portrait image material, and determine R image materials suitable for image-to-video processing and / or S video materials that need to be processed by video augmentation from the M multimedia materials.

[0306] In some embodiments, the material analysis module 101 may be used to execute some or all of the contents of the aforementioned S602, S701 to S706, S1001, and S1102, which will not be elaborated here.

[0307] In some embodiments, the material analysis module 101 can send the material information of M materials to the template matching module 102, send the determined R image materials to the image-to-video module 103, send the determined S video materials to the video expansion module 104, and send the MN multimedia materials other than the N multimedia materials (R image materials and / or S video materials) to the preprocessing module 105.

[0308] In some embodiments, the material analysis module 101 may also send the material information of R image materials to the image-to-video module 103, and / or send the material information of S video materials to the video extension module 104.

[0309] In some embodiments, the material analysis module 101 can determine the material information of each material based on the aforementioned material analysis model.

[0310] The template matching module 102 is used to determine one or more video templates (e.g., the first video module) that match the M multimedia materials based on the material information of the M multimedia materials, and send the determined one or more video templates to the synthesis module 106.

[0311] In some embodiments, the template matching module 102 may also determine the video template that matches the M multimedia materials among multiple video templates based on the material information of M multimedia materials, R video data, and S video data.

[0312] In some embodiments, the template matching module 102 can be used to perform the content of determining the first video template in the aforementioned S604, S606, S1001 to S1004, S1103, which will not be elaborated here.

[0313] The image-to-video module 103 is used to obtain R video data based on R image materials. For example, the image-to-video module 103 can input each image material and the first prompt information of each image material into the image-to-video model to obtain the video data corresponding to each image material.

[0314] For example, in some embodiments, the image-generated video module 103 can be used to perform the aforementioned steps S604, S801 to S803, and S1102 regarding the determination of R video data, which will not be elaborated here.

[0315] The video expansion module 104 is used to obtain S video data based on S video clips. For example, the video expansion module 104 can input each video clip and the second prompt information of each video clip into the video expansion model to obtain the video data corresponding to each video clip.

[0316] For example, in some embodiments, the video expansion module 104 can be used to perform the content of determining S video data in the aforementioned S604, S901 to S903, and S1102, which will not be elaborated here.

[0317] The preprocessing module 105 is used to preprocess the MN multimedia materials. For example, the preprocessing module 105 can determine the composition information of the image materials in the MN multimedia materials and the highlight information of the video materials in the MN multimedia materials.

[0318] In some embodiments, the preprocessing module 105 can send the determined composition information and highlight information to the compositing module. For example, the preprocessing module 105 can be used to perform the preprocessing of MN multimedia materials in the aforementioned S603.

[0319] The compositing module 106 is used to combine N video data and MN multimedia materials into a first video template based on N video data corresponding to N multimedia materials, MN multimedia materials, composition information of image materials in MN multimedia materials, and highlight information of video materials in MN multimedia materials, to obtain a first video.

[0320] In some embodiments, the synthesis module 106 may be used to execute part or all of the aforementioned S606 and S1103, which will not be elaborated here.

[0321] It should be noted that, Figure 12 The structure of the video generation apparatus 100 shown is only an example. In other embodiments, the video generation apparatus 100 may include more or fewer modules, or merge or split some modules, without limitation.

[0322] It should be noted that, Figure 12 The modules in the video generation apparatus 100 shown can be deployed on the same electronic device or on multiple electronic devices; this embodiment does not limit this. For example, the material analysis module 101, template matching module 102, image-generated video module 103, video expansion module 104, preprocessing module 105, and compositing module 106 can all be deployed on the first electronic device. As another example, the material analysis module 101, preprocessing module 105, and compositing module 106 can be deployed on the first electronic device, while the template matching module 102, image-generated video module 103, and video expansion module 104 can be deployed on a second electronic device. Yet another example: the material analysis module 101, preprocessing module 105, and compositing module 106 can be deployed on the first electronic device, while the template matching module 102, image-generated video module 103, and video expansion module 104 can be deployed on other electronic devices different from the first electronic device.

[0323] The following describes a software architecture suitable for the first electronic device.

[0324] For example, Figure 13 A schematic diagram of a software architecture is shown according to some embodiments of this application.

[0325] refer to Figure 13 The operating system of the first electronic device can adopt a layered structure, consisting of the following layers from bottom to top: kernel layer, system service layer, framework layer, and application layer. Layers communicate with each other through software interfaces. System functions can be tailored, added, or combined at the subsystem level depending on the deployment scenario of different device forms, and each subsystem can also be tailored, added, or combined at the functional level.

[0326] The application layer can include applications in the first electronic device, such as system applications and extended / third-party applications. System applications can include gallery, desktop, control bar, settings, contacts, phone, camera, etc., while extended / third-party applications can include social applications, travel applications, etc.

[0327] In some embodiments, a gallery can be used to perform the video generation method provided in the embodiments of this application. For example, a gallery application can be used to display multimedia materials in a first electronic device and edit M multimedia materials selected by the user into a first video based on the video generation method provided in the foregoing embodiments.

[0328] For example, the image library can call the model in the AI ​​business subsystem to obtain the material information of M multimedia materials, determine the aforementioned N multimedia materials among the M multimedia materials based on the obtained material information, call the model in the AI ​​business subsystem (or call the model in the cloud (e.g., a second electronic device) through the interface provided by the AI ​​business subsystem) to convert the N multimedia materials into N video data through the first processing, determine the first video template among multiple video templates, and synthesize the N video data and the aforementioned M and N multimedia materials into the first video template to obtain the first video.

[0329] The framework layer provides application programming interfaces (APIs) and programming frameworks for applications in the application layer. For example, the framework layer may include ArkUI, user application frameworks, and capability frameworks (an capability is a lightweight application; the capability framework schedules and manages the operation and lifecycle of capabilities).

[0330] The HarmonyOS API is a series of open capabilities provided to support HarmonyOS application development. The HarmonyOS API can be set at the framework layer or independently of the framework layer. The HarmonyOS API includes the Audio API (audio service), Push API (push service), and Account API (account service), among others.

[0331] The system service layer comprises the core capabilities of the system, providing services to applications through the framework layer. The system service layer may include a set of basic system capability subsystems, a set of basic software service subsystems, a set of enhanced software service subsystems, and a set of hardware service subsystems.

[0332] The system's basic capability subsystem set provides fundamental capabilities for the operation, scheduling, and migration of distributed applications across multiple devices. This set may include distributed soft bus, distributed data management, distributed task scheduling, and Ark multi-language runtime; it may also include multi-modal input subsystem, graphics subsystem, security subsystem, and AI subsystem.

[0333] In some embodiments, the AI ​​subsystem can provide AI capabilities to applications and services in the first electronic device. For example, the aforementioned material analysis model, prompt word enhancement model, image-generated video model, and video augmentation model can be integrated into the AI ​​subsystem or the AI ​​subsystem can provide calling interfaces to the application layer or system model, so that applications (e.g., image libraries) can call the above models to implement the video generation method provided in the embodiments of this application.

[0334] Basic software service subsystem set: provides public and general software services; the basic software service subsystem set may include event notification subsystem, telephone service subsystem, multimedia subsystem, etc.

[0335] Enhanced software service subsystem suite: Provides differentiated enhanced software services for different devices; the enhanced software service subsystem suite may include smart screen proprietary business subsystem, wearable proprietary business subsystem, Internet of Things proprietary business subsystem, etc.

[0336] Basic software service subsystem set: provides hardware services; the hardware service subsystem set may include location service subsystem, unified user identity and access management subsystem, wearable proprietary hardware service subsystem, biometric identification, Internet of Things proprietary hardware service subsystem, etc.

[0337] Distributed task scheduling enables distributed service management (discovery, synchronization, registration, and invocation), supporting remote startup, remote invocation, remote connection, and migration of applications across devices.

[0338] Distributed data management enables data synchronization, data storage, data sharing, and data access across all scenarios and devices.

[0339] The distributed soft bus provides communication-related capabilities for seamless interconnection between multiple devices, including: wireless fidelity service capabilities, Bluetooth service capabilities, StarFlash service capabilities, soft bus, inter-process communication and other communication capabilities.

[0340] Ark Multilingual Runtime is a unified compilation runtime platform designed to support the joint compilation and execution of multiple programming languages ​​and multiple chip platforms.

[0341] The kernel layer can include the memory abstraction layer, kernel subsystem, driver subsystem, etc.

[0342] The kernel abstraction layer (KAL) provides basic kernel capabilities to upper layers by shielding the differences between multiple kernels, including but not limited to process / thread management, memory management, file system, network management, and peripheral device management.

[0343] Kernel subsystem: Supports selecting the appropriate kernel for different resource-constrained devices, not limited to... kernel, Kernel (HongMeng kernel), lightweight operating system Kernel, etc.

[0344] Driver Subsystem: The driver framework is the foundation for the open system hardware ecosystem, providing unified peripheral access capabilities and a framework for driver development and management. The driver framework includes: display drivers, camera drivers, audio drivers, Bluetooth drivers, sensor drivers, etc.

[0345] It should be noted that, Figure 13 The software architecture shown is only an example. In other embodiments, the software architecture of the first electronic device may include more or fewer modules, may merge or split parts of the model, or may adopt other architectures, which are not limited here.

[0346] This application provides a computer program product that, when executed on an electronic device, enables the electronic device to implement the video generation methods provided in the foregoing embodiments.

[0347] This application also provides a computer-readable storage medium storing one or more programs / instructions, which, when executed by an electronic device, enable the electronic device to implement the video generation method provided in the foregoing embodiments.

[0348] This application also provides an electronic device, which includes at least one memory and one or more processors. The memory is used to store one or more programs / instructions, and the at least one processor is used to execute one or more programs / instructions stored in the memory to enable the electronic device to implement the video generation method provided in the foregoing embodiments.

[0349] further, Figure 14According to some embodiments of this application, a structural schematic diagram of a mobile phone 10 is shown.

[0350] like Figure 14 As shown, the mobile phone 10 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0351] Processor 110 may include one or more processing units, such as a central processing unit (CPU), application processor, modem processor, graphics processing unit (GPU), image signal processor (ISP), microcontroller unit (MCU), video codec, digital signal processor (DSP), baseband processor, neural network processing unit (NPU), field-programmable gate array (FPGA), etc. In some embodiments, different processing units may be independent devices or integrated into one or more processors.

[0352] The processor 110 may also include a memory for storing one or more programs and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store one or more programs or data that the processor 110 has just used or is reusing. When the processor 110 needs to use the one or more programs or data again, it can retrieve them directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0353] In some embodiments, the processor 110 can be used to execute one or more programs / instructions corresponding to the video generation methods provided in the foregoing embodiments. For example, the processor 110 can use an NPU or GPU to run the aforementioned material analysis model to obtain material information of various multimedia materials, use an NPU or GPU to run a graph-to-video model to generate R video data, use an NPU or GPU to run a video augmentation model to generate S video data, use an NPU or GPU to run a cue word enhancement model to obtain first cue information corresponding to image materials and second cue information corresponding to video materials, and use the CPU and / or NPU and / or GPU to synthesize a first video, etc.

[0354] The charging management module 140 receives charging input from the charger. While charging the battery 142, the charging management module 140 can also supply power to the electronic device through the power management module 141.

[0355] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to power the processor 110, internal memory 121, display 194, camera 193, and wireless communication module 160, etc.

[0356] The wireless communication function of mobile phone 10 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor.

[0357] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals.

[0358] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, for use in the mobile phone 10.

[0359] The wireless communication module 160 can provide solutions for wireless communication applications on the mobile phone 10, including wireless local area networks (WLAN) (such as Wi-Fi), Bluetooth (BT), global navigation satellite system (GNSS), near field communication (NFC), infrared (IR) technology, etc.

[0360] For example, in some embodiments, mobile phone 10 can send material information of M multimedia materials and N image materials to other electronic devices through mobile communication module 150 and / or wireless communication module 160, and mobile phone 10 can receive N video data and a first video template from other electronic devices through mobile communication module 150 and / or wireless communication module 160.

[0361] The mobile phone 10 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0362] The display screen 194 is used to display images. For example, the display screen 194 can be used to display the interfaces in the foregoing embodiments.

[0363] Camera 193 is used to capture still images or videos.

[0364] The external memory interface 120 can be used to connect an external memory card. The external memory card communicates with the processor 110 through the external memory interface 120 to perform data storage functions. For example, application data can be stored on the external memory card.

[0365] The internal memory 121 can be used to store one or more programs and corresponding data. The internal memory 121 may include a program storage area and a data storage area. In some embodiments, the program storage area may store the operating system, at least one application required for a function, such as programs / instructions corresponding to the video generation methods provided in the foregoing embodiments, such as the aforementioned image-generated video model, material analysis model, video augmentation model, and prompt enhancement model. The data storage area may store data created during the use of the mobile phone 10, such as the aforementioned N video data sets. Furthermore, the internal memory 121 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, general-purpose flash memory, etc. The processor 110 executes various functional applications of the mobile phone 10 by running one or more programs stored in the internal memory 121 and / or one or more programs stored in memory disposed in the processor 110.

[0366] The phone 10 can achieve audio functions, such as music playback, through an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, and an application processor.

[0367] Audio module 170 is used to convert digital audio information into analog audio signal output, and also to convert analog audio input into digital audio signal. Audio module 170 can also be used for encoding and decoding audio signals.

[0368] The loudspeaker 170A, also known as a "loudspeaker", is used to convert audio electrical signals into sound signals.

[0369] The receiver 170B, also known as the "earpiece", is used to convert audio electrical signals into sound signals.

[0370] The microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals.

[0371] The 170D headphone jack is used to connect wired headphones.

[0372] The 180E accelerometer can detect the magnitude of acceleration in various directions (typically three axes) of the mobile phone 10. When the mobile phone 10 is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the device posture of electronic devices.

[0373] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch buttons. Mobile phone 10 can receive button input and generate key signal inputs related to user settings and function control of mobile phone 10.

[0374] Motor 191 can generate vibration as a notification. Indicator 192 can be an indicator light, used to indicate charging status, power changes, or to indicate messages, notifications, etc.

[0375] The SIM card interface 195 is used to connect a SIM card or eSIM.

[0376] It is understood that the structure of the mobile phone 10 shown in the embodiments of this application does not constitute a specific limitation on the electronic device. In other embodiments of this application, the electronic device may include more or fewer components than shown, or combine some components, or split some components, or have different component arrangements. The components shown may be implemented in hardware, software, or a combination of software and hardware.

[0377] It should be noted that the first electronic device and the second electronic device may adopt the same, similar or different structure as mobile phone 10, which will not be elaborated here.

[0378] It should be noted that the first electronic device, second electronic device, third electronic device, or electronic device in the embodiments of this application can be any type of electronic device, such as a mobile station (MS), an electronic device (MT), a computer, etc., and the embodiments of this application do not limit this. For example, electronic devices can be mobile phones, smart TVs, tablets, desktop computers, laptops, servers, computing nodes, virtual reality (VR) devices, augmented reality (AR) devices, terminals in industrial control, terminals in self-driving cars, terminals in smart cities, terminals in smart homes, and in-vehicle terminals (such as in-vehicle systems, smart cockpits, in-vehicle tablets, and in-vehicle projectors), etc.

[0379] It should be noted that in the embodiments of this application, "or" describes the relationship between related objects, indicating that there can be two relationships. For example, A or B can mean either A or B, where A or B can be singular or plural.

[0380] It should be noted that in the embodiments of this application, terms such as "for example," "in some embodiments," "in another embodiment," "in other embodiments," and "exemplarily" are used to indicate examples, illustrations, or descriptions. Any embodiment or design scheme described as an "example" in this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the term "example" is intended to present concepts in a concrete manner.

[0381] It should be noted that the terms "first" and "second" used in the embodiments of this application are used only for the purpose of distinguishing descriptions and should not be construed as indicating or implying relative importance or order.

[0382] It should be noted that the term "equal to" in the embodiments of this application can be used with "greater than" and is applicable to the technical solutions adopted when "greater than". It can also be used with "less than" and is applicable to the technical solutions adopted when "less than". It should be noted that when "equal to" is used with "greater than", it is not used with "less than"; when "equal to" is used with "less than", it is not used with "greater than".

[0383] In the accompanying drawings, some structural or methodological features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be necessary. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. Furthermore, the inclusion of structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may be omitted or may be combined with other features.

[0384] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method of video generation, the method comprising: The method applied to a first electronic device comprises: receiving a first instruction, wherein the first instruction indicates to cut M multimedia clips into a video, the M multimedia clips comprising at least one image clip and / or at least one video clip, and M is an integer greater than 1; obtaining N video data, wherein the N video data is obtained based on first processing on N multimedia clips in the M multimedia clips, N is an integer greater than or equal to 1 and less than or equal to M; generating a first video based on the N video data and M-N multimedia clips in the M multimedia clips, wherein the M-N multimedia clips are multimedia clips other than the N multimedia clips in the M multimedia clips; wherein, in a case that the N multimedia clips comprise a first image clip and the N video data comprises first video data corresponding to the first image clip, the first processing comprises processing of generating the first video data based on the first image clip; and / or, in a case that the N multimedia clips comprise a first video clip and the N video data comprises second video data corresponding to the first video clip, the first processing comprises processing of generating the second video data based on the first video clip, and a video duration of the second video data is greater than a video duration of the first video clip.

2. The method of claim 1, wherein, The N multimedia clips comprise an image clip satisfying a first condition in the at least one image clip, and / or a video clip satisfying a second condition in the at least one video clip.

3. The method of claim 2, wherein, The first condition comprises at least one of the following conditions: the image clip is an image clip selected by a user; at least one subject in the image clip is a preset subject; a scene label in the image clip matches a preset scene label; the image clip comprises a portrait, and a size of a face in the portrait is greater than or equal to a preset size; the image clip comprises the portrait, and a degree of occlusion of the face in the portrait is less than or equal to a first degree threshold; the image clip comprises the portrait, and a degree of occlusion of the portrait in the image clip is less than or equal to a second degree threshold; the image clip comprises the portrait, and a clarity degree of the face in the portrait is greater than or equal to a clarity degree threshold; the image clip comprises the portrait, and a shooting angle corresponding to the portrait is within a preset angle range.

4. The method according to claim 2 or 3, characterized in that, The second condition comprises at least one of the following conditions: the video clip is a video clip selected by a user; a video duration of the video clip is less than or equal to a preset duration; at least one subject in the video clip is a preset subject; a video quality of the video clip is less than or equal to a video quality threshold.

5. The method according to any one of claims 1 to 4, characterized in that, The first video data is generated by the first electronic device based on the first image clip by calling a first model, or the first video data is received by the first electronic device from a second electronic device after the first electronic device sends the first image clip to the second electronic device. And / or, the second video data is generated by the first electronic device calling a second model based on the first video material, or the second video data is received by the first electronic device from a second electronic device after sending the first video material to the second electronic device.

6. The method of claim 5, wherein, The first video data is generated by the first electronic device or the second electronic device calling the first model based on the first image material and first prompt information; And / or the second video data is generated by the first electronic device or the second electronic device calling the second model based on the first video material and second prompt information.

7. The method of claim 6, wherein, The first prompt information includes first prompt text input by a user and / or second prompt text generated based on the M multimedia materials; And / or, the second prompt information includes third prompt text input by a user and / or fourth prompt text generated based on the M multimedia materials.

8. The method according to any one of claims 1 to 7, characterized in that, The video duration of the first video data is determined based on the video duration of at least part of the at least one video material, or the video duration of the first video data is a preset duration.

9. The method according to any one of claims 1 to 8, characterized in that, The first video is generated based on N video data and M-N multimedia materials from the M multimedia materials, including: A first video template is used to generate a first video based on the N video data and the M-N multimedia materials.

10. The method of claim 9, wherein, The first video template is a video template whose at least one template parameter matches the N video data and the M-N multimedia materials from the M multimedia materials, or a video template selected by a user from a plurality of video templates, wherein the template parameters include at least one of template size, template direction, template semantic label, template scene label, number of material slots, slot duration of material slots, and slot type of material slots.

11. The method of claim 10, wherein, The first video template satisfies a third condition, wherein the third condition includes at least one of the following conditions: The template scene label of the first video template matches the scene label of the M multimedia materials; The number of material slots of the first video template is greater than or equal to M; The template direction of the first video template matches the material direction of the M multimedia materials; The slot duration of the first material slot with the maximum slot duration among the material slots of the first video template is less than or equal to the video duration of each video material in the at least one video material; The number of material slots with slot duration less than a first duration in the material slots of the first video template is greater than or equal to N, and the first duration is the video duration of each video data in the N video data; The distribution of slot types of the material slots of the first video template matches the material types of the N video data and the M-N multimedia materials.

12. The method of claim 11, wherein, The first video template is a video template with the maximum similarity of semantic labels to the semantic labels of the M multimedia materials among one or more video templates in the plurality of video templates that satisfy the third condition.

13. The method according to any one of claims 10 to 12, characterized in that, The first video template is received by the first electronic device from a third electronic device, or determined by the first electronic device from the plurality of video templates.

14. A method of video generation, the method comprising: The method comprises: The first electronic device receives a first instruction, wherein the first instruction indicates to cut M multimedia materials into a video, the M multimedia materials comprising at least one image material or at least one video material, and M is an integer greater than 1; The first electronic device sends N multimedia materials in the M multimedia materials to a second electronic device, N being an integer greater than or equal to 1 and less than or equal to M; The second electronic device respectively performs first processing on the N multimedia materials to obtain N video data, and sends the N video data to the first electronic device, wherein in the case that the N multimedia materials comprise a first image material, the first processing comprises processing of generating first video data based on the first image material, and / or in the case that the N multimedia materials comprise a first video material, the first processing comprises processing of generating second video data based on the first video material, the video duration of the second video data being greater than the video duration of the first video data; The first electronic device generates a first video based on the N video data and M-N multimedia materials in the M multimedia materials, wherein the M-N multimedia materials are multimedia materials in the M multimedia materials other than the N multimedia materials.

15. A readable storage medium, characterized by, The readable storage medium comprises one or more programs, and the one or more programs, when executed on an electronic device, cause the electronic device to implement the method in any one of claims 1 to 13.

16. A computer program product, characterised in that, The computer program product, when running on an electronic device, causes the electronic device to implement the method in any one of claims 1 to 13.

17. An electronic device, comprising: Comprise: One or more memories for storing one or more programs; One or more processors for executing the one or more programs to cause the electronic device to implement the method in any one of claims 1 to 13.

Images