Video generation method, device, electronic device and storage medium

Abstract

The present disclosure discloses a video generating method, an apparatus, an electronic device, and a storage medium. The video generating method includes: receiving a first trigger operation for generating a video using a target music (S101); in response to the first trigger operation, selecting a candidate material that matches the target music from a candidate material set of the user according to first music feature information of the target music as a target candidate material, and generating a target video according to the target candidate material and with the target music as background music (S102).

Description

[Technical field]

[0001] This application claims priority to a Chinese patent application bearing application number 202110057581.2, filed with the China Patent Office on January 15, 2021, the entire contents of which are incorporated herein by reference.

[0002] The present disclosure relates to the field of computer technology, for example, to video generation methods, devices, electronic devices, and storage media. [Background technology]

[0003] Short video applications generally provide users with the ability to create and post short videos.

[0004] When creating a video, a user usually needs to manually shoot or select video materials, set the sequence of multiple video materials, set appropriate background music for the selected video materials, and then use a short video application to composite the selected materials and background music into the video.

[0005] However, in related technologies, the hurdles for creating short videos are high, it takes a long time to complete the creation, and the user's creation experience is poor. Summary of the Invention [Problem to be solved by the invention]

[0006] The present disclosure provides a video generation method, device, electronic device, and storage medium for reducing the hurdles to creating short videos and shortening the time required to create short videos. [Means for solving the problem]

[0007] The present disclosure relates to Receiving a first trigger operation to generate a video using a target music; In response to the first trigger operation, a candidate material matching the target music is selected from a candidate material set of the user based on the first music feature information of the target music, and the candidate material is set as a target candidate material; and a target video is generated according to the target candidate material and has the target music as background music. A video generation method is provided.

[0008] The present disclosure relates to a first receiving module configured to receive a first trigger operation to generate a video using a target music; a video generating module configured to, in response to the first trigger operation, select candidate material matching the target music from a candidate material set of the user based on first music feature information of the target music, as a target candidate material, and generate a target video according to the target candidate material and with the target music as background music; A video generating device is also provided.

[0009] The present disclosure relates to one or more processors; and a memory configured to store one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the video generation method. An electronic device is also provided.

[0010] The present disclosure relates to A computer-readable storage medium having a computer program stored thereon, When the program is executed by a processor, the program realizes the video generation method. A computer readable storage medium is also provided. [Brief description of the drawings]

[0011] [Figure 1] 1 is a flowchart of a video generation method according to an embodiment of the present disclosure. [Diagram 2] FIG. 2 is a schematic diagram of a music details page according to an embodiment of the present disclosure. [Diagram 3] 13 is a schematic diagram of a material notification window according to an embodiment of the present disclosure. [Figure 4] FIG. 13 is a schematic diagram of an authority release window according to an embodiment of the present disclosure. [Diagram 5] 1 is a flowchart of another video generation method according to an embodiment of the present disclosure. [Figure 6] FIG. 2 is a schematic diagram of a loading page according to an embodiment of the present disclosure. [Figure 7] FIG. 1 is a schematic diagram of a preview page according to an embodiment of the present disclosure. [Figure 8] FIG. 13 is a schematic diagram of a template window according to an embodiment of the present disclosure. [Figure 9] FIG. 2 is a schematic diagram of a material window according to an embodiment of the present disclosure. [Figure 10] 1 is a schematic diagram of a material editing window according to an embodiment of the present disclosure. [Figure 11] FIG. 2 is a schematic diagram of a video editing page according to an embodiment of the present disclosure. [Figure 12] FIG. 1 is a block diagram illustrating a configuration of a video generating device according to an embodiment of the present disclosure. [Figure 13] FIG. 1 is a structural schematic diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, the embodiments of the present disclosure will be described in more detail with reference to the drawings. Although the drawings show some embodiments of the present disclosure, the present disclosure can be realized in various forms and should not be construed as being limited to the embodiments described herein, but rather these embodiments are provided for a clearer and more complete understanding of the present disclosure. The drawings and embodiments of the present disclosure are merely illustrative.

[0013] The steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. Also, method embodiments may include additional steps and / or steps shown omitted from execution. The scope of the present disclosure is not limited in this respect.

[0014] As used herein, the term "comprising" and variations thereof are open inclusive, i.e., "including, but not limited to." The term "based on" means "based at least in part on." The term "one embodiment" refers to "at least one embodiment." The term "another embodiment" refers to "at least one other embodiment." The term "some embodiments" refers to "at least some embodiments." Relevant definitions of other terms are provided below in the description.

[0015] It should be noted that the concepts of "first", "second", etc. referred to in this disclosure are merely intended to distinguish between devices, modules, or units, and are not intended to limit the order or interdependence of functions performed by these devices, modules, or units.

[0016] It should be noted that the modifications "one" and "multiple" referred to in the present disclosure are exemplary but not limiting, and one of ordinary skill in the art will appreciate that they should be understood as "one or more" unless the context clearly indicates otherwise.

[0017] The names of messages or information exchanged between multiple devices in the embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0018] FIG. 1 is a flowchart of a video generation method according to an embodiment of the present disclosure. The method can be performed by a video generation device, where the device can be realized by software and / or hardware, and can be configured in an electronic device such as a mobile phone or a tablet computer. The video generation method according to an embodiment of the present disclosure is applied to a scene of automatically generating a video, in particular, a scene of automatically generating a music video. As shown in FIG. 1, the video generation method according to the embodiment may include the following steps:

[0019] In S101, a first trigger operation for generating a video using a target music is received.

[0020] The first trigger operation can be understood as a trigger operation for instructing an electronic device to automatically generate a video using one piece of music, such as a trigger operation of clicking a video generation control, which can be displayed on a page such as a music list page, a music detail page, a music playback page, and / or a video playback page. Correspondingly, the target music is a music that a user wants to use as background music to create a video, such as a music in a selected state on a music list page when the user executes the first trigger operation, a music corresponding to a music detail page, a music currently being played on a music playback page, or a background music used for a video currently being played on a video playback page, etc. Hereinafter, the first trigger operation will be described as an example of a trigger operation acting on a video generation control on a music detail page.

[0021] In this embodiment, as shown in FIG. 2, the music details page of one piece of music may display music information of the music (e.g., music thumbnail 20 of the music, music name, user name of the poster, number of plays, etc.) and video generation control 21, whereby a user triggers the video generation control 21 to automatically generate a video with the music as background music, for example, to instruct the electronic device to generate a music video with the music as background music. A play / pause control 22, a favorite control 23, and / or a same type shooting control 24 may be set, whereby a user can click the play / pause control 22 to instruct the electronic device to play / pause the music, click the favorite control 23 to register the music as a favorite, and / or click the same type shooting control 24 to instruct the electronic device to switch the current display page from the music details page of the music to a video shooting page and turn on the camera, so that the user can shoot a video with the music as background music. Additionally, when a user enters the music details page for the first time, the music details page may display presentation information 25 about the video generation control 21, such as presentation videos, photos and / or text, etc., so that the user can clarify the function of the video generation control 21.

[0022] For example, the electronic device displays a music detail page for a piece of music, and when a user wants to instruct the electronic device to automatically generate a video using the music, the user can click a video generation control displayed on the music detail page. Correspondingly, when the electronic device monitors that the user has clicked the video generation control on the music detail page, it can determine that the music corresponding to the music detail page is a target music, and determine that a first trigger operation for generating a video using the target music has been received.

[0023] When the user clicks on the video generation control for the first time and / or when the user has not released permission to access the user's material, the electronic device may display a corresponding window to inform the user to read the user's material and / or to prompt the user to release permission to access the material, in which case the first triggering operation may be an operation to trigger a corresponding control in the displayed window.

[0024] As shown in Fig. 2, the electronic device displays a music detail page of one piece of music, and when it monitors that the user clicks the video generation control 21 in the music detail page, it can determine whether the user clicks the video generation control 21 for the first time. When the user clicks for the first time, it displays a material notification window 30 as shown in Fig. 3, and the material notification window 30 informs the user that the user's material will be read when automatically generating a video. When it monitors that the user clicks the close control 31 in the material notification window 30, it stops displaying the material notification window 30. When it monitors that the user clicks the start experience control 32 in the material notification window 30, it determines whether the user has released the authority to access the material. Correspondingly, when it is not the user's first click, it can directly determine whether the user has released the authority to access the material. As a result, if it is determined that the user has released the authority to access the material, it can be confirmed that the first trigger operation has been received, and if it is determined that the user has not released the authority to access the material, an authority release window 40 can be displayed as shown in Figure 4, to inform the user that the authority to access the material has been released. Furthermore, if it is monitored that the user has clicked on the allow control 41 in the authority release window 40, it is confirmed that the first trigger operation has been received, and if it is monitored that the user has clicked on the disallow control 42 in the authority release window 40, the display of the authority presentation window 40 is stopped.

[0025] In S102, in response to the first trigger operation, a candidate material matching the target music is selected from a candidate material set of the user based on the first music feature information of the target music, and the candidate material is set as a target candidate material. A target video is generated according to the target candidate material, with the target music as background music.

[0026] The first music characteristic information may be music characteristic information of the target music, such as lyrics information, rhythm information, and / or music style information of the target music. The target video may be understood as a video generated by the electronic device using the target music. The candidate material may be understood as a material that the electronic device can use to generate the target video, and may be all materials in the user's local album and / or cloud album, or a part of materials in the user's local album and / or cloud album, such as materials with high shooting quality in the user's local album and / or cloud album, which may include photos and / or videos, etc. In one embodiment, the materials include photos and videos that the user can obtain in advance by taking, downloading, importing, screenshotting, or recording.

[0027] When the electronic device receives the first trigger operation, it obtains a candidate material set of the user, and based on the first music feature information of the target music, according to a preset selection rule, selects candidate materials that match the target music from the candidate material set as target candidate materials, and synthesizes all the selected target candidate materials into the target video, so that the user can automatically generate the target video by selecting the target music. For example, when a user watches a video, if the user likes the background music in the video, the user can switch the current display page to the video detail page of the background music and click the video generation control in the video detail page to generate a video with the background music as the background music. The user does not need to select a video template and video materials, which further reduces the difficulty of video creation and shortens the time it takes for the user to create a video. Here, the selection rules for the target candidate material can be set by the developer as needed. For example, according to the first music feature information of the target music, a candidate material whose screen style is similar to the first music feature information can be selected as the target candidate material; or a video template whose second music feature information of the background music matches the first music feature information of the target music can be selected as the target video template, and a candidate material whose material theme is similar to the template body of the target video template can be selected as the target candidate material; this embodiment is not limited thereto.

[0028] In one embodiment, in order to improve the quality of the generated target video, the candidate materials in the candidate material set may be materials with good shooting quality in the user's local album and / or cloud album, for example, before selecting candidate materials that match the target music from the user's candidate material set as target candidate materials, the method further includes obtaining materials in the user's albums, including a local album and / or an associated cloud album, and sorting multiple materials from the albums according to the aesthetic scores of the materials to form the user's candidate material set.

[0029] The related cloud album can be understood as the cloud album related to the account that the user logs in. The album from which the user obtains material may include the user's local album and / or the user's related cloud album, and if there is no user's related cloud album or the user has not opened the permission to access the cloud album, the album may only include the local album, i.e., the electronic device may only obtain the material in the user's local album. If there is a user's related cloud album and the user has opened the permission to access the cloud album, the album may include the local album and the related cloud album, i.e., the electronic device may obtain the material in the user's local album and the material in the user's related cloud album.

[0030] For example, the electronic device can first obtain all materials in the user album, aesthetically evaluate all materials according to preset aesthetic evaluation rules or using a pre-trained aesthetic evaluation model, obtain aesthetic scores for all materials, and then select materials whose aesthetic scores are greater than a preset score threshold as candidate materials, or select a preset number (e.g., 30 or 50) of materials in descending order of aesthetic score as candidate materials, thereby constituting a set of candidate materials for the user.

[0031] In the above embodiment, the timing at which the electronic device retrieves materials in the user album and constructs a candidate material set according to the aesthetic scores of all materials can be flexibly set as necessary, for example, the electronic device may perform the above operation when it receives a first trigger operation, or may perform the above operation when the launch of the application to which the target music belongs is completed and a preset time (e.g., 10 s) after the launch of the application to which the target music belongs is completed, or when it is monitored that the user has turned on the camera of the electronic device by the application to which the target music belongs.

[0032] The video generating method according to the present embodiment includes receiving a first trigger operation for generating a video using a target music, and in response to the first trigger operation, selecting a candidate material that matches the target music from a candidate material set of the user according to the first music feature information of the target music as a target candidate material, and generating a target video with the target music as background music according to the target candidate material. By adopting the above technical solution, the present embodiment can automatically generate a video using a material that matches the music selected by the user, thereby reducing the hurdle of video creation, shortening the time it takes for the user to create a video, and increasing the degree of matching between the generated video and the music selected by the user, thereby improving the quality of the generated video.

[0033] 5 is a flowchart of another video generating method according to an embodiment of the present disclosure. The embodiment of the present embodiment can be combined with one or more preferred embodiments of the above-mentioned embodiment. Preferably, selecting candidate material that matches the target music from the candidate material set of the user according to the first music feature information of the target music as the target candidate material includes: obtaining second music feature information of the background music of each video template, selecting a video template whose second music feature information matches the first music feature information of the target music as the target video template, and determining the target candidate material according to the relevance between the material theme of each candidate material in the candidate material set of the user and the template theme of the target video template.

[0034] Preferably, the target video template includes a plurality of material slots for filling with materials, and after determining target candidate materials based on the relevance between the material subject of each candidate material in the user's candidate material set and the template subject of the target video template, the method further includes sequentially obtaining target candidate materials that satisfy the filling conditions of each material slot of the target video template as materials waiting to be filled in the material slots, and filling the materials waiting to be filled into the material slots, and generating a target video according to the target candidate materials includes generating a target video according to the target video template after filling the materials waiting to be filled.

[0035] Preferably, the method further includes: after filling the material slots with the material to be filled, displaying a preview page, and displaying the material to be composited in the target video template on the preview page, including the material to be filled. Preferably, the method further includes: after displaying the preview page, receiving a second trigger operation acting on a template control on the preview page, displaying a template window displaying template information of a plurality of video templates in response to the second trigger operation, receiving a third trigger operation acting on the template information displayed in the template window, and switching the target video template to a video template to which the template information corresponding to the third trigger operation belongs, and updating the material to be composited in the target video template in response to the third trigger operation.

[0036] Correspondingly, as shown in FIG. 5, the video generating method according to this embodiment includes the following steps.

[0037] In S201, a first trigger operation for generating a video using a target music is received.

[0038] In S202, in response to the first trigger operation, second music feature information of the background music of each video template is obtained, and a video template whose second music feature information matches the first music feature information of the target music is selected as a target video template.

[0039] The video template can be understood as a template for creating a video. The second music feature information can be music feature information of background music that needs to be added to the video template, such as rhythm information and / or music style information of the background music, etc.

[0040] Exemplarily, when the electronic device receives a first trigger operation, it can switch the current display page from a music details page to a loading page, obtain the music feature information of the background music of each video template, and select a video template whose music feature information of the background music matches the music feature information of the target music as the target video template, as shown in FIG. 6 .

[0041] In this embodiment, as shown in FIG. 6 , the loading page may display the current loading progress and a cancel control 60, allowing a user to trigger the cancel control 60 to instruct the electronic device to stop performing subsequent operations and switch the current display page from the loading page to the music details page of the target music. The manner of determining the target video template can be set according to needs. For example, when determining the target video template, the electronic device may directly use a preset matching degree calculation rule or use a pre-trained matching degree calculation model to calculate the matching degree between the second music feature information of the background music of each video template in the template library and the first music feature information of the target music, and select the video template with the highest matching degree as the target video template; or, first, determine whether or not there is a video using a video template when generating a video with the target music as background music among the videos posted by the user who performs the first trigger operation and / or other users, and if there is a video using a video template when generating a video with the target music as background music, obtain the video template used in the video and set it as the target video template; if there is no video using a video template when generating a video with the target music as background music, calculate the matching degree between the second music feature information of the background music of each video template in the template library and the first music feature information of the target music, and select the video template with the highest matching degree as the target video template.

[0042] In addition, when there are multiple templates used in a video with the target background music as background music, or when there are multiple video templates with the highest degree of matching, one video template may be randomly selected as the target video template, or a video template that best matches the user's personalized information, such as the video style of a video posted and / or played by the user, may be selected as the target video template. Alternatively, when determining the target video template, a preset number (e.g., 5 or 10) of video templates may be selected according to the degree of matching between the music feature information of the background music of the video with the target music as background music or each template and the music feature information of the target music, and the video template that best matches the user's personalized information may be selected from the preset number of video templates as the target video template. This allows the generated target video to be further tailored to the user's personalized needs, improving the user's experience.

[0043] At S203, target candidate material is determined based on a relationship between the material subject matter of each candidate material in the user's set of candidate material and the template subject matter of the target video template.

[0044] The material subject can be understood as the subject of the material, for example, the subject of the candidate material in the candidate material set, and can be determined according to the screen content of the material. The template subject can be understood as the subject of the video template, for example, the subject of the target video template, and the subject of one material / video template can be, for example, plants, buildings, scenery, people, travel, or food, etc.

[0045] In this embodiment, based on the template theme of the target video template, materials in the candidate material set whose material theme is similar to the template theme of the target video template (for example, the degree of matching is greater than a preset threshold value) may be selected as target candidate materials, or the candidate material set may be divided into a plurality of sub-candidate material sets with different material themes according to the material theme of each candidate material in advance, so that after the target video template is determined, candidate materials in the sub-candidate material set whose material theme is similar to the template theme of the target video template can be selected as target candidate materials. Here, when selecting target candidate materials, all candidate materials whose material theme is similar to the template theme of the target video template may be determined as target candidate materials, or a preset number (for example, 20 or 50, etc.) of candidate materials whose material theme is similar to the template theme of the target video template may be selected as target candidate materials, and this embodiment is not limited thereto.

[0046] In S204, target candidate materials that satisfy the filling conditions of each material slot of the target video template are sequentially obtained and set as materials waiting to be filled in the material slots, and the materials waiting to be filled are filled into the material slots, where the target video template includes multiple material slots for filling with materials.

[0047] The fill conditions may include the aspect ratio of the material that needs to be filled (eg, 16:9, etc.), the type of material (eg, video or photo, etc.) and / or the maximum duration of the material.

[0048] Taking the filling condition as an example where the filling condition is the aspect of the material, after determining the target candidate material, the electronic device may select one material slot of the target video template as the current material slot, for example, select the first material slot of the target video template as the current material slot, select one target candidate material that meets the aspect ratio requirement of the current material slot from the target candidate materials that have not been determined as materials waiting to be filled, and set it as material waiting to be filled for the current material slot, fill the current material slot with the material waiting to be filled, determine the next material slot of the target candidate material as the current material slot, and return to the operation of selecting from the target candidate materials that have not been determined as materials waiting to be filled until there is no next material slot.

[0049] In this step, if there is only one target candidate material that satisfies the filling condition of the current material slot, the target candidate material can be determined as the material waiting to be filled in the material slot, if there are multiple target candidate materials that satisfy the filling condition of the current material slot, one video type target candidate material can be randomly selected from the multiple target candidate materials to be the material waiting to be filled in the current material slot in order to improve the quality of the generated target video, if there is no video type target candidate material from the multiple target candidate materials, one target candidate material can be randomly selected to be the material waiting to be filled in the current material slot. Also, if the material waiting to be filled is a video type target candidate material whose duration is longer than the maximum duration of the material that can be filled in the current material slot, when filling the material waiting to be filled in the current material slot, a key video whose duration matches the maximum duration can be extracted from the material waiting to be filled, and the extracted key video can be filled in the current material slot.

[0050] If the loading fails, for example, if the target video template cannot be loaded normally due to a network issue, or if the number of materials whose material subject is similar to the template subject of the target video template is too small (e.g., less than the number of material slots contained in the target video template), or if there is no target candidate material that meets the filling conditions of one or more material slots, the electronic device can stop the loading, switch the current display page from the loading page to the music details page of the target music, and present the user with the fact that the loading failed, for example, presenting the user with a message such as "Video generation failed. Please try again later" or "Material is insufficient, please try again after refilling", so as to clarify the cause of the video generation failure for the user.

[0051] In S205, a preview page is displayed, and the preview page displays the ready-to-be-composited materials in the target video template, including the ready-to-be-filled materials.

[0052] The material to be composited can be understood as material (e.g., video or photos) in the target video template that the user composites into the target video, and may include material to be filled in each material slot in the target video template, and may further include scene change material, which may be carried in the target video template, or which may be generated based on adjacent material to be filled in and filled between adjacent material to be filled in after the electronic device fills the material to be filled in the target video template.

[0053] For example, after the loading of the target video template and the materials corresponding to each material slot of the target video template is completed, as shown in FIG. 7, the current display page can be switched from a loading page to a preview page, and the materials waiting to be synthesized in the target video template are displayed on the preview page, for example, all the materials waiting to be synthesized are displayed sequentially according to the order of all the materials waiting to be synthesized in the target video template, so as to clarify the materials for the user to generate the target video and the arrangement order of all the materials, or the materials waiting to be synthesized currently displayed on the preview page are switched according to the user's switching operation.

[0054] At S206, a second trigger operation acting on a template control on the preview page is received.

[0055] In S207, in response to the second trigger operation, a template window in which template information of a plurality of video templates is displayed is displayed.

[0056] The second trigger operation may be an operation for triggering a template control displayed on the preview page, such as a click operation acting on the template control. The template information of the video template may be related information of the video template, such as a template thumbnail and / or a template name of the video template. The template window can be used to display a template list, and the template list may include all the video templates in the template library. Only templates in which the matching degree between the second music characteristic information of the background music and the first music characteristic information of the target music is greater than a set matching degree threshold may be included, and this embodiment is not limited thereto.

[0057] In this embodiment, as shown in Fig. 7, the preview page may be set with a template control 70 for the user to instruct the electronic device to display a template window, so that the user can click the template control 70 if he / she wants to change the target video template used when generating the target video. Correspondingly, when the electronic device monitors that the user clicks the template control 70, it can determine that a second trigger operation is received, and in response to the second trigger operation, it displays a template window 80 and displays template information of each video template in the template window 80, for example, as shown in Fig. 8, displays a template thumbnail and / or template name of each video template in the template window. Here, each video template may be displayed in the template window 80 randomly or in order of the degree of matching between the second music characteristic information of the background music and the first music characteristic information of the target music.

[0058] At S208, a third trigger operation acting on the template information displayed in the template window is received.

[0059] In S209, in response to the third trigger operation, the target video template is switched to a video template to which template information corresponding to the third trigger operation belongs, and the material to be synthesized in the target video template is updated.

[0060] The third trigger operation can be understood as an operation acting within the display area of ​​the template information of one video template in the template window, such as an operation of clicking on the template thumbnail of one video template displayed in the template window.

[0061] As shown in Fig. 8, when a user wants to switch a target video template to another video template, the user can click the thumbnail of the video template displayed in the template window 80. Correspondingly, when the electronic device monitors that the user clicks the template thumbnail of a video template displayed in the video template, it can determine that a third trigger operation is received, switch the target video template to the video template, and determine the target candidate template of the video template according to the association between the material subject of each candidate material in the user's candidate material set and the template subject of the video template, sequentially obtain the target candidate materials that meet the filling conditions of each material slot of the video template as the material waiting to be filled in the material slot, fill the material waiting to be filled in each material slot into the corresponding material slot, and update the material waiting to be composited displayed on the preview page to the material waiting to be composited in the video template so that the user can view the updated material waiting to be composited. And when the electronic device receives the third trigger operation, it can display a download presentation window 81 in response to the trigger operation to present the template currently being downloaded to the user. In this case, when the updated material waiting to be composited is displayed on the preview page, the display of the presentation window 81 may be stopped so as not to prevent the user from viewing the updated material waiting to be composited.

[0062] Also, still referring to Fig. 8, after switching the target video template, a first save control 82 and a first cancel control 83 may be displayed on the preview page, so that when it is monitored that the user clicks the first save control 82, the switched target video template and the material waiting to be composited can be saved, and the template window 80 can be closed, as shown in Fig. 7. When it is monitored that the user clicks the first cancel control 83, the switched target video template and the material waiting to be composited can be switched back to the previous target video template and the material waiting to be composited, and the template window 80 can be closed, as shown in Fig. 7.

[0063] In one embodiment, still referring to FIG. 7, a material control 71 may be displayed on the preview page, and when a user first enters the preview page, introductory information 72 about the function of the material control 71 may be displayed. If a user wants to replace the material waiting to be filled that is filled in the target video template, the user may click on the material control 71. Correspondingly, when the electronic device monitors that the user clicks on the material control 71, the electronic device may display a material window 90, as shown in FIG. 9, in which material information of the user's material, candidate material or target candidate material, such as a material thumbnail, is displayed. A second save control 91 and a second cancel control 92 may be displayed on the preview page.

[0064] Thus, when a user wishes to replace the material currently displayed on the preview page with another material, the user can click on a thumbnail of the other material displayed in the material window 90. When the electronic device monitors that the user has clicked on a material thumbnail of one material displayed in the material window 90, if the material is a photograph, the material displayed on the preview page at the current time can be replaced with the photograph. When the material is a video, as shown in FIG. 10, the video material editing window 100 can be displayed so that the user can select a portion of the video to use as the material. When the electronic device monitors that the user has clicked on a confirmation control 102 in the video material editing window 100, the material displayed on the preview page at the current time can be replaced with the portion selected by the user. When the electronic device monitors that the user has clicked on a cancel control 101 in the video material editing window 100, the portion selected by the user is cancelled. When the electronic device monitors that the user has clicked on a material exchange control 103 in the video material editing window 100, the video material editing window 100 is closed and the material window 90 is displayed so that the user can exchange the material.

[0065] If the user wants to save the modifications to the material, he can click the second save control 91, and when the electronic device monitors that the user has clicked the second save control 91, it can save the modified material waiting to be filled and close the material window 90. If the user wants to cancel the modifications to the material, he can click the second cancel control 92, and when the electronic device monitors that the user has clicked the second cancel control 92, it can switch the modified material waiting to be composited to the previous material waiting to be composited and close the material window 90, as shown in FIG.

[0066] The material window 90 may also display a text editing control 93 for the user to edit the text carried in the target template. When the electronic device fills the target video template with the material waiting to be filled, the electronic device can synchronously fill the text carried in the target video template that needs to be displayed in the video with the corresponding photo or video plate, and when the electronic device monitors that the user clicks on the text editing control 93 in the material window 90, it displays a text editing window for the user to edit the text carried in the target video template that is added to the photo or video plate.

[0067] In S210, a target video is generated according to the target video template after filling the to-be-filled material.

[0068] Illustratively, the electronic device can automatically generate a target video based on the filled target video template after each material slot of the target video template is filled with the material to be filled, for example, can composite all the material to be filled / material to be composited into the target video according to the order of all the material to be filled / material to be composited into the target video after the material slots of the target video template are filled with the material to be filled. After the filling of all the material slots of the target video template is completed, a preview page is displayed so that the user can view and modify the target video template and the material to be filled / material to be composited into the target video when it is monitored that the user triggers a video composite control on the preview page, for example, generating a target video according to the target video template after filling with the material to be filled includes using the target music as background music, and compositing all the material to be composited into the target video according to the order of all the material to be composited into the target video when a fourth trigger operation acting on the video composite control on the preview page is received.

[0069] Illustratively, continuing to refer to FIG. 7, the electronic device displays a video composition control 73 on the preview page, and if the user wants to compose the material to be composed in the target video template into the target video, the user can click the video composition control 73. Correspondingly, when the electronic device monitors that the user clicks the video composition control 73, the electronic device can determine that a fourth trigger operation is received, and in response to the fourth trigger operation, the electronic device can set the target music as background music, compose all the material to be composed into the target video according to the order of all the material to be composed in the target video template, and switch the current display page from the preview page to a video editing page, so that the user can edit the composited target video, such as replacing the background music of the target video, adding text, stickers, effects and / or filters to the target video, and correcting the image quality of the target video, as shown in FIG. 11. And when the electronic device monitors that the user clicks the stories control 110 on the video editing page, the electronic device can post the target video and set the target video to be viewed only within a preset number of days (e.g., one day or one week). When it is monitored that the user clicks the next control 111 on the video editing page, the currently displayed page is switched from the video editing page to the posting page so that the user can edit the title, posting position and public range of the target video, and the target video is posted or the target video is saved and stored in the draft. Here, if the material to be synthesized is a video type material, the audio data of the material to be synthesized may be deleted or the material to be synthesized may be muted in order to avoid sounds other than the target music suddenly appearing or disappearing in the obtained target video during synthesis.

[0070] Also, as shown in FIG. 7, a first return control 74 may be displayed on the preview page, whereby if the user wants to return to the music details page of the target music, the user can click on the first return control 74. In response, when the electronic device monitors that the user has clicked on the first return control 74, the electronic device can display a first save prompting window (not shown in FIG. 7) to present to the user that the current effect will be lost if the user returns. In response, when the electronic device monitors that the user has clicked on the first confirmation control in the first save prompting window, the electronic device switches the currently displayed page from the preview page to the music details page of the target music. As shown in FIG. 11, a second return control 112 may be displayed on the video editing page. In response, when the user wants to return to the preview page, the electronic device can display a second save prompting window (not shown in FIG. 11) to present to the user that the edited content in the video editing page will not be saved. In response, when the electronic device monitors that the user has clicked on the second return control 112, the electronic device can display a second save prompting window (not shown in FIG. 11) to present to the user that the edited content in the video editing page will not be saved. In response, when the electronic device monitors that the user has clicked on the second confirmation control in the second save prompting window, the electronic device switches the currently displayed page from the video editing page to the preview page.

[0071] The video generation method of this embodiment selects a target video template according to the music characteristics of the target video, determines target candidate materials according to the theme of the target video template, and synchronously updates the materials filled in the target video template after the user changes the target video template, not only can it automatically generate a video according to the music selected by the user, but also can improve the quality of the generated target video, make the generated target video more in line with the user's intention, and improve the user's usage experience.

[0072] 12 is a block diagram of a video generating device according to an embodiment of the present disclosure. The device can be realized by software and / or hardware, and can be configured in an electronic device such as a mobile phone or a tablet computer, and can generate a video by executing a video generating method. As shown in FIG. 12, the video generating device according to this embodiment may include a first receiving module 1201 and a video generating module 1202.

[0073] The first receiving module 1201 is configured to receive a first trigger operation for generating a video using a target music, and the video generating module 1202 is configured to respond to the first trigger operation by selecting a candidate material matching the target music from a candidate material set of the user based on first music characteristic information of the target music, as a target candidate material, and generating a target video with the target music as background music according to the target candidate material.

[0074] The video generating device of this embodiment receives a first trigger operation for generating a video using a target music through a first receiving module 1201, and in response to the first trigger operation, the video generating module 1202 selects a candidate material matching the target music from the candidate material set of the user according to the first music feature information of the target music as a target candidate material, and generates a target video with the target music as background music according to the target candidate material. By adopting the above technical solution, this embodiment can automatically generate a video using a material matching the music selected by the user, reducing the hurdle of video creation, shortening the time it takes for the user to create a video, and increasing the degree of matching between the generated video and the music selected by the user, thereby improving the quality of the generated video.

[0075] In the above embodiment, the video generation module 1202 may include a template acquisition unit configured to acquire second music feature information of the background music of each video template, and select a video template whose second music feature information matches the first music feature information of the target music as a target video template, and a material determination unit configured to determine a target candidate material based on the association between the material theme of each candidate material in a user's candidate material set and the template theme of the target video template.

[0076] In the above embodiment, the target video template may include a plurality of material slots for filling with material, and the video generation module 1202 may further include a material filling unit configured to determine target candidate materials based on the association between the material subject of each candidate material in the user's candidate material set and the template subject of the target video template, and then sequentially obtain target candidate materials that satisfy the filling conditions of each material slot of the target video template as materials waiting to be filled in the material slots, and fill the materials waiting to be filled in the material slots, and a video generation unit configured to generate a target video according to the target video template after the materials waiting to be filled have been filled.

[0077] In the above embodiment, the video generation module 1202 may further include a material display unit configured to display a preview page after filling the material slots with the material waiting to be filled, and to display the material waiting to be composited including the multiple materials waiting to be filled in the target video template on the preview page.

[0078] In the above embodiment, the video generation module 1202 may further include a first receiving unit configured to receive a second trigger operation acting on a template control in the preview page after displaying the preview page, a template display unit configured to display a template window in which template information of a plurality of video templates is displayed in response to the second trigger operation, a second receiving unit configured to receive a third trigger operation acting on the template information displayed in the template window, and a material updating unit configured to switch the target video template to a video template to which template information corresponding to the third trigger operation belongs, and update material waiting to be synthesized in the target video template in response to the third trigger operation.

[0079] In the above embodiment, when the video generation unit receives a fourth trigger operation acting on a video composition control on the preview page, the video generation unit may be configured to use the target music as background music, and compose all of the materials waiting to be composed into the target video according to the order of all the materials waiting to be composed in the target video template.

[0080] The video generation device of this embodiment may further include a material acquisition module configured to acquire materials in the user's albums, including local albums and / or associated cloud albums, before selecting candidate materials matching the target music from the user's candidate material set as target candidate materials, and a material selection module configured to select a plurality of materials from the albums according to aesthetic scores of the materials to form the user's candidate material set.

[0081] The video generating device according to the embodiment of the present disclosure can execute the video generating method according to any embodiment of the present disclosure, and has functional modules and effects corresponding to the execution of the video generating method. For technical details not described in detail in this embodiment, please refer to the video generating method according to any embodiment of the present disclosure.

[0082] Hereinafter, referring to FIG. 13, a structural schematic diagram of an electronic device (e.g., a terminal device) 1300 suitable for implementing an embodiment of the present disclosure is shown. The terminal device in the embodiment of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, personal digital assistants (PDA), tablet computers (PAD), portable media players (PMP), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital (Television, TV), desktop computers, and the like. The electronic device shown in FIG. 13 is merely an example and does not impose any limitations on the functions and scope of use of the embodiment of the present disclosure.

[0083] As shown in Fig. 13, the electronic device 1300 may include a processing unit (e.g., a central processor, a graphic processor, etc.) 1301, which can execute various appropriate operations and processes based on a program stored in a read-only memory (ROM) 1302 or loaded from a storage device 1308 into a random access memory (RAM) 1303. The RAM 1303 further stores various programs and data necessary for the operation of the electronic device 1300. The processing unit 1301, the ROM 1302, and the RAM 1303 are connected to each other via a bus 1304. An input / output (I / O) interface 1305 is also connected to the bus 1304.

[0084] Typically, input devices 1306 including, for example, a touch panel, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc., output devices 1307 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc., storage devices 1306 including, for example, a magnetic tape, hard disk, etc., and communication devices 1309 can be connected to the I / O interface 1305. The communication devices 1309 enable the electronic device 1300 to communicate wirelessly or wired with other devices to exchange data. Although FIG. 13 illustrates the electronic device 1300 with various devices, it should be understood that it is not necessary to implement or include all of the illustrated devices. Instead, more or fewer devices may be implemented or included.

[0085] According to an embodiment of the present disclosure, the process described with reference to the flowchart above may be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product including a computer program carried on a non-transitory computer-readable medium, the computer program including program code for executing the method shown in the flowchart. In such an embodiment, the computer program may be downloaded from a network by the communication device 1309 and installed, or may be installed from the storage device 1306, or may be installed from the ROM 1302. When the computer program is executed by the processing device 1301, it performs the above-mentioned functions defined in the method of the embodiment of the present disclosure.

[0086] It should be noted that the computer readable medium described in this disclosure may be a computer readable signal medium, a computer readable storage medium, or any combination of the above. The computer readable storage medium may be, for example, but is not limited to, an electric, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. Further specific examples of the computer readable storage medium may include, but are not limited to, an electrical connection having one or more leads, a portable computer disk, a hard disk, a RAM, a ROM, an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this disclosure, the computer readable storage medium may be any tangible medium that contains or stores a program that can be used in or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, in which computer-readable program code is carried. Such a propagated data signal may take a variety of forms, including, but not limited to, an electromagnetic signal, an optical signal, or any suitable combination of the above. A computer-readable signal medium may be any computer-readable medium other than a computer-readable storage medium, which may transmit, propagate, or transmit a program for use in or in conjunction with an instruction execution system, apparatus, or device.The program code contained in the computer readable medium may be transmitted over any suitable medium, including, but not limited to, electrical wire, optical cable, radio frequency (RF), or the like, or any suitable combination of the above.

[0087] In some embodiments, the clients and servers may communicate using any now known or later developed network protocol, such as HyperText Transfer Protocol (HTTP), and may interconnect with any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include a Local Area Network (LAN), a Wide Area Network (WAN), a network off a network (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), and any now known or later developed network.

[0088] The computer-readable medium may be included in the electronic device, or may exist separately and not attached to the electronic device.

[0089] The computer-readable medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device receives a first trigger operation for generating a video using a target music, and in response to the first trigger operation, selects a candidate material that matches the target music from a candidate material set of the user based on first music characteristic information of the target music, sets it as a target candidate material, and generates a target video with the target music as background music according to the target candidate material.

[0090] Computer program code for carrying out the operations of the present disclosure can be written in one or more programming languages ​​or combinations thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and even conventional procedural programming languages ​​such as "C" or similar. The program code may run entirely on the user's computer, partially on the user's computer, as a single, separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or business server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a LAN or WAN, or may be connected to an external computer (e.g., connected via the Internet using an Internet Service Provider).

[0091] The flowcharts and block diagrams in the drawings illustrate possible architectures, functions and operations of the methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowcharts or block diagrams may represent a module, program or part of code, which includes one or more executable instructions for implementing a certain logical function. It should be noted that in some alternative implementations, the functions described in the blocks may occur in a different order than the order described in the drawings. For example, two blocks shown as connected may actually be executed substantially in parallel or in reverse order depending on the functions involved. It should be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts may be implemented in a system based on dedicated hardware that executes a certain function or operation, or may be implemented in a combination of dedicated hardware and computer instructions.

[0092] The units according to the embodiments of the present disclosure may be implemented in a software or hardware manner, and the names of the modules, if any, do not limit the units themselves.

[0093] The functionality of the present disclosure may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Parts (ASSPs), System on Chip (SOCs), Complex Programmable Logic Devices (CPLDs), etc.

[0094] In the present specification, a machine-readable medium may be a tangible medium that can contain or store a program used in or in conjunction with an instruction execution system, apparatus or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the above. Further specific examples of machine-readable storage media include an electrical connection by one or more wires, a portable computer disk, a hard disk, a RAM, a ROM, an EPROM or flash memory, a CD-ROM, an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0095] According to one or more embodiments of the present disclosure, Example 1 comprises: Receiving a first trigger operation to generate a video using a target music; In response to the first trigger operation, a candidate material matching the target music is selected from a candidate material set of the user based on the first music feature information of the target music, and the candidate material is set as a target candidate material; and a target video is generated according to the target candidate material and has the target music as background music. A video generation method is provided.

[0096] According to one or more embodiments of the present disclosure, Example 2 is based on the method of Example 1, and includes: selecting a candidate material that matches the target music from a candidate material set of a user according to first music feature information of the target music as a target candidate material; Obtain second music feature information of the background music of each video template, and select a video template whose second music feature information matches the first music feature information of the target music as the target video template; determining target candidate material based on an association between a material subject of each candidate material in the user's set of candidate material and a template subject of the target video template.

[0097] According to one or more embodiments of the present disclosure, Example 3 is based on the method of Example 2, wherein the target video template includes a plurality of material slots for filling with material; after determining target candidate material based on an association between a material subject of each candidate material in the user's candidate material set and a template subject of the target video template, The method further includes: sequentially acquiring target candidate materials that satisfy a filling condition of each material slot of the target video template, and setting the target candidate materials as materials waiting to be filled in the material slots; and filling the material slots with the materials waiting to be filled; generating a target video in accordance with the candidate target material; generating the target video according to a target video template after filling the to-be-filled material.

[0098] According to one or more embodiments of the present disclosure, Example 4 is based on the method of Example 3, after the waiting material is filled into the material slot, The method further includes displaying a preview page, and displaying, on the preview page, material ready to be composited including the material ready to be filled in the target video template.

[0099] According to one or more embodiments of the present disclosure, Example 5 is based on the method of Example 4, after displaying the preview page, receiving a second trigger operation acting on a template control on the preview page; displaying a template window in response to the second trigger operation, the template window displaying template information of a plurality of video templates; receiving a third trigger operation acting on the template information displayed in the template window; The method further includes, in response to the third trigger operation, switching the target video template to a video template to which template information corresponding to the third trigger operation belongs, and updating a material to be synthesized in the target video template.

[0100] According to one or more embodiments of the present disclosure, Example 6 is based on the method according to Example 4 or 5, wherein generating the target video according to a target video template after filling the to-be-filled material includes: When a fourth trigger operation acting on a video composition control on the preview page is received, the target music is used as background music, and all the materials to be composed are composed into the target video according to the order of all the materials to be composed in the target video template.

[0101] According to one or more embodiments of the present disclosure, Example 7 is based on the method according to any one of Examples 1 to 5, and before selecting candidate material that matches the target music from the candidate material set of the own user as target candidate material, Obtaining material in the user's albums, including local albums and / or associated cloud albums; The method further includes filtering a plurality of materials from the album according to their aesthetic scores to form a candidate set of materials for the user.

[0102] According to one or more embodiments of the present disclosure, Example 8 comprises: a first receiving module configured to receive a first trigger operation to generate a video using a target music; a video generating module configured to, in response to the first trigger operation, select candidate material matching the target music from a candidate material set of the user based on first music feature information of the target music, as a target candidate material, and generate a target video according to the target candidate material and with the target music as background music; A video generating device is provided.

[0103] According to one or more embodiments of the present disclosure, Example 9 is one or more processors; a memory configured to store one or more programs; An electronic device comprising: When the one or more programs are executed by the one or more processors, the one or more processors implement the video generation method of any one of Examples 1 to 7. Provide electronic devices.

[0104] According to one or more embodiments of the present disclosure, Example 10 provides a computer-readable storage medium having a computer program stored thereon, the computer-readable storage medium realizing the video generation method described in any one of Examples 1 to 7 when the program is executed by a processor.

[0105] Although operations have been described in a particular order, it should not be understood that these operations must be performed in the particular order shown, or in a forward order. In certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although the above discussion includes specific implementation details, these should not be understood as limiting the scope of the disclosure. Some features described in the description of a single embodiment may be implemented in combination in a single embodiment. Conversely, various features described in the description of a single embodiment may be implemented in multiple embodiments alone or in any suitable subcombination.

Claims

1. Receiving a first trigger operation to generate a video using a target piece of music; In response to the first trigger operation, a candidate material matching the target music is selected from a candidate material set of the user based on the first music feature information of the target music, and the candidate material is set as a target candidate material, and a target video is generated according to the target candidate material and has the target music as background music. Selecting a candidate material that matches the target music from a candidate material set of the user based on the first music feature information of the target music as a target candidate material; Obtain second music feature information of background music of each video template, and select a video template whose second music feature information matches the first music feature information of the target music as the target video template; determining target candidate material based on an association between a material subject of each candidate material in the user's set of candidate material and a template subject of the target video template; Video generation methods.

2. the target video template includes a plurality of material slots for filling with material, and after determining target candidate material based on a relationship between a material subject of each candidate material in the user's candidate material set and a template subject of the target video template, The method further includes: sequentially acquiring target candidate materials that satisfy a filling condition of each material slot of the target video template, and setting the target candidate materials as materials waiting to be filled in the material slots; and filling the material slots with the materials waiting to be filled; generating a target video in accordance with the candidate target material; generating the target video according to a target video template after filling the to-be-filled material; The method of claim 1.

3. After the waiting material is filled into the material slot, displaying a preview page, and displaying, on the preview page, a composite-ready material including the fill-ready material in the target video template; The method of claim 2.

4. After displaying the preview page, receiving a second trigger operation acting on a template control in the preview page; displaying a template window in which template information of a plurality of video templates is displayed in response to the second trigger operation; receiving a third triggering operation acting on the template information displayed in the template window; In response to the third trigger operation, the target video template is switched to a video template to which template information corresponding to the third trigger operation belongs, and a material to be synthesized in the target video template is updated. The method according to claim 3.

5. generating the target video according to the target video template after filling the to-be-filled material, and when receiving a fourth trigger operation acting on a video composition control on the preview page, using the target music as background music, and compositing all the materials to be composed into the target video according to the order of all the materials to be composed in the target video template. The method according to claim 3 or 4.

6. before selecting candidate material that matches the target music from the candidate material set of the user as a target candidate material; Obtaining material in a user's albums, including at least one of a local album and an associated cloud album; and filtering a plurality of materials from the album according to aesthetic scores of the materials to form a candidate set of materials for the user.

5. The method according to any one of claims 1 to 4.

7. a first receiving module configured to receive a first trigger operation to generate a video using a target music; a video generating module configured to, in response to the first trigger operation, select candidate material matching the target music from a candidate material set of the user based on first music feature information of the target music, as a target candidate material, and generate a target video according to the target candidate material and with the target music as background music; Selecting a candidate material that matches the target music from a candidate material set of the user based on the first music feature information of the target music as a target candidate material; Obtain second music feature information of background music of each video template, and select a video template whose second music feature information matches the first music feature information of the target music as the target video template; determining target candidate material based on an association between a material subject of each candidate material in the user's set of candidate material and a template subject of the target video template; Video generation equipment.

8. At least one processor; and a memory configured to store at least one program, When said at least one program is executed by said at least one processor, said at least one processor implements the video generation method according to any one of claims 1 to 6. electronic equipment.

9. A computer-readable storage medium having a computer program stored thereon, The computer program, when executed by a processor, implements the video generation method according to any one of claims 1 to 6. A computer-readable storage medium.

Images