Video generation method and device
By obtaining feature information of video reference keyframes and material images, and combining artificial intelligence technology, videos with special effects, sound and style are generated, the problem of single video elements in the existing technology is solved, and convenient personalized video shooting is achieved.
Patent Information
- Application Number
- CN202411352988.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-26
- Publication Date
- 2024-12-03
AI Technical Summary
In the prior art, the video elements are relatively single during video shooting, and personalized video cannot be easily captured and cannot meet the personalized needs of users.
By obtaining image feature information of reference keyframes in the video, combining material images, videos with special effects, sound and style information are generated, and artificial intelligence technology is used to integrate the shooting effects and environmental atmosphere in the video template to generate videos with personal style.
It realizes convenient shooting of personalized videos, integrates the shooting effects and environmental atmosphere in the video templates, avoids obvious editing traces, and meets the personalized needs of users.
Smart Images

Figure CN119071570A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of image processing technology, and specifically relates to a video generation method and device. Background Art
[0002] Users can shoot videos with different styles on the video shooting platform. For example, in the short video application, when shooting a video, if a portrait is detected, makeup effects of different character styles can be added to the face area, hat effects of different styles can be added above the face area, and hair accessories effects of different styles can be added above the face area. However, the characters in the video after adding special effects are dull and the editing traces are obvious.
[0003] For another example, in a short video application, when shooting a video, if a portrait is detected, the face area is placed in a certain scene. This scene is just a background image, which makes the video display effect poor and cannot meet the user's personalized needs.
[0004] It can be seen that in the prior art, during the video shooting process, some synthesized video elements are relatively simple, which makes it impossible to conveniently shoot personalized videos. Summary of the invention
[0005] The purpose of the embodiment of the present application is to provide a video generation method that can conveniently shoot personalized videos.
[0006] In a first aspect, an embodiment of the present application provides a video generation method, the method comprising: obtaining first image feature information of a reference key frame in a first video; receiving a first input of a material image; in response to the first input, obtaining a material key frame in the material image that matches the reference key frame according to the first image feature information; generating a second video according to second image feature information, the material key frame and the material image; wherein the second image feature information includes at least: special effects information of the first video, sound information of the first video, and style information of the first video.
[0007] In a second aspect, an embodiment of the present application provides a video generating device, comprising: a first acquisition module, used to acquire first image feature information of a reference key frame in a first video; a receiving module, used to receive a first input of a material image; a second acquisition module, used to respond to the first input and, based on the first image feature information, acquire a material key frame in the material image that matches the reference key frame; a generating module, used to generate a second video based on the second image feature information, the material key frame and the material image; wherein the second image feature information includes at least: special effects information of the first video, sound information of the first video, and style information of the first video.
[0008] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps described in the first aspect are implemented.
[0009] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.
[0010] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.
[0011] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.
[0012] In an embodiment of the present application, first, the first image feature information of the reference key frame in the first video is obtained. The first video can be a related video selected by the user based on the current shooting environment. Here, the first video can be used as a video template. Then, the user uploads the material image, which can be an image shot by the user in the current shooting environment, so as to obtain the material key frame matching the reference key frame in the material image by using the first image feature information of the reference key frame of the first video; further, the second image feature information of the first video is referred to, and the extracted material key frame and the material image itself are combined to generate the second video. Among them, the second image feature information is not limited to the special effect information, sound information and style information of the first video. It can be seen that in an embodiment of the present application, in the shooting scene, the user can select a high-quality video such as a video template on the Internet, so as to integrate some shooting effects, the environmental atmosphere presented, etc. in the video template into the video shot by the user in combination with artificial intelligence technology, so that some video effects are added to the shot video, but there is no obvious editing trace, thereby helping the user to easily shoot a personalized video with a personal style. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] Figure 1 is a flowchart of a video generation method in some embodiments of the present application;
[0014] Figure 2 is one of the display schematic diagrams of the electronic device in some embodiments of the present application;
[0015] Figure 3This is a second display schematic diagram of an electronic device in some embodiments of the present application;
[0016] Figure 4 is a schematic diagram illustrating the display of an electronic device in some embodiments of the present application;
[0017] Figure 5 is a block diagram of a video generating device according to some embodiments of the present application;
[0018] Figure 6 Schematic diagram of the hardware structure of the electronic device of some embodiments of the present application;
[0019] Figure 7 It is a schematic diagram of the hardware structure of an electronic device of some embodiments of the present application. DETAILED DESCRIPTION
[0020] The following will be combined with the drawings of the embodiments of the present application to clearly describe the technical solutions of the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present application belong to the scope of protection of this application.
[0021] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally a class, and the number of objects is not limited. For example, the target object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated before and after are in an "or" relationship.
[0022] The video generation method provided in the embodiment of the present application is described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.
[0023] like Figure 1 FIG. 1 is a flowchart of a video generation method according to an embodiment of the present application. The method is applied to an electronic device as an example. The method includes:
[0024] Step 110: Obtain first image feature information of a reference key frame in the first video.
[0025] The scenario to which this embodiment is applied is, for example, a user sees a video in a short video application, in which the protagonist in the video is walking on the beach, and there is also atmospheric background music and character dialogue in the video. The overall atmosphere of the video is very good, and the user also wants to shoot such a video. Therefore, the user can first save the video as the first video in this step.
[0026] Optionally, the user saves a video in the album in advance, such as Figure 2 As shown, the user clicks on the upload template video option 201 in the interface of a video production application, thereby displaying multiple videos in the album. The user selects a video and clicks on the Finish option, and the video is the first video.
[0027] Optionally, this embodiment is implemented by an artificial intelligence (AI) big model, that is, the AI big model processes the first video and subsequent material images to generate a second video.
[0028] Among them, the AI big model can be a model that comes with the system and can realize artificial intelligence processing; the AI big model can also be a model in the server that can realize artificial intelligence processing. Therefore, data transmission is required between the user end and the server end to make the generated video displayed on the user end.
[0029] For example, the AI big model is the Blue Heart big model and the Chat Generative Pre-trained Transformer (ChatGPT) model.
[0030] Optionally, the AI big model performs AI analysis on the first video to generate a video template. Figure 2 As shown, multiple video templates generated based on history are displayed in a list form on the user side. The user clicks on any video template option 202 to enter Figure 3 The template details interface shown is where users can create videos using corresponding video templates.
[0031] Based on the above content, in this step, in the process of generating a video template, first, a key frame in the first video is obtained to serve as a reference key frame of the video template.
[0032] For example, the AI big model processes all frames of the first video using relevant algorithms. If the calculated composition score of a frame is greater than a preset composition score threshold, the frame is used as a reference key frame.
[0033] For another example, the AI big model processes all frames of the first video using relevant algorithms. If the calculated portrait score of a certain frame is greater than a preset portrait score threshold, the frame is used as a reference key frame.
[0034] Furthermore, after obtaining the reference key frames in the first video, the image feature information of each reference key frame is extracted. For the convenience of distinction, the image feature information of the reference key frame is defined as the first image feature information.
[0035] Optionally, the first image feature information is used to characterize shooting content such as a shooting object, a shooting environment, etc. in the reference key frame.
[0036] Step 120: Receive a first input of a material image.
[0037] In some embodiments of the present application, the above-mentioned first input is used for the user to upload the material image to the AI big model, thereby triggering the AI big model to perform AI analysis on the material image, and generate the second video in combination with the second image feature information of the first video. The first input can be a first operation. Exemplarily, the above-mentioned first input includes but is not limited to: the user's touch input to the controls on the screen or the screen area through a touch device such as a finger or a stylus, or a voice command input by the user, or a specific gesture input by the user, or other feasible inputs, which can be determined according to actual use requirements, and the embodiments of the present invention are not limited. The specific gesture in the embodiment of the present application can be any one of a single-click gesture, a sliding gesture, a drag gesture, a pressure recognition gesture, a long press gesture, an area change gesture, a double-press gesture, and a double-click gesture; the click input in the embodiment of the present application can be a single-click input, a double-click input, or any number of click inputs, etc., and can also be a long press input or a short press input.
[0038] For example, the first input may be Figure 3 In the template details interface shown, the user clicks on the import material option 301 to display a list of images in the album, and the user can select at least one image in the image list as a material image.
[0039] Optionally, the type of the material image is a picture or a video; the number of the material images is not limited.
[0040] For example, if the user hopes to shoot a similar effect to the first video, he can first imitate the first video to shoot a video in a similar shooting environment, and the shot video can be used as the material image; or, the user can also imitate the first video to shoot multiple photos, and the multiple photos taken can be used as the material images.
[0041] An application scenario includes a user going to the beach and wanting to shoot a video of the sunset at the beach. The user can browse and search in various short video applications in advance to find a video that he or she is satisfied with as the first video, and then refer to the video content in the first video, find a suitable position, imitate the actions of the characters in the video, such as walking, and shoot a video, thereby using the shot video as a material image.
[0042] Step 130: In response to the first input, a material key frame matching the reference key frame in the material image is obtained according to the first image feature information.
[0043] In this step, a frame matching the reference key frame is obtained in the material image according to the first image feature information as the material key frame.
[0044] Among them, the matching here can be understood as a high similarity.
[0045] Step 140: Generate a second video according to the second image feature information, the material key frame and the material image, wherein the second image feature information at least includes: special effect information of the first video, sound information of the first video, and style information of the first video.
[0046] Specifically, the special effect information of the first video includes special effect information of the reference key frame.
[0047] After determining the material key frame, at least the special effect information of the reference key frame, the sound information of the first video, and the style information of the first video can be used to process the material key frame and the material image to generate a second video.
[0048] For example, in the first video, the protagonist and the puppy are walking in the sunset, the background music is soothing, the protagonist is talking to the puppy, and in a certain shot, the protagonist's face is close-up to highlight the protagonist's comfortable state, so that the whole video creates a lazy and leisurely life scene; in the second video, the user and his puppy are also walking in the sunset, the same background music can appear, the user and his puppy are talking the same words, and in a certain shot, the user's face can also be close-up.
[0049] For reference, when the material image is a video, the video can be processed to generate a second video based on the video; or, when the material image includes multiple pictures, a video can be generated based on the multiple pictures, and each picture in the video can appear for a certain period of time in the order imported by the user, or the material key frame can appear for a longer period of time, thereby generating a second video on this basis.
[0050] like Figure 4As shown, in the video preview window 401, when the second video is generated based on the automatic processing of the large model, it is played in the video preview window 401.
[0051] In an embodiment of the present application, first, the first image feature information of the reference key frame in the first video is obtained. The first video can be a related video selected by the user based on the current shooting environment. Here, the first video can be used as a video template. Then, the user uploads the material image, which can be an image shot by the user in the current shooting environment, so as to obtain the material key frame matching the reference key frame in the material image by using the first image feature information of the reference key frame of the first video; further, the second image feature information of the first video is referred to, and the extracted material key frame and the material image itself are combined to generate the second video. Among them, the second image feature information is not limited to the special effect information, sound information and style information of the first video. It can be seen that in an embodiment of the present application, in the shooting scene, the user can select a high-quality video such as a video template on the Internet, so as to integrate some shooting effects, the environmental atmosphere presented, etc. in the video template into the video shot by the user in combination with artificial intelligence technology, so that some video effects are added to the shot video, but there is no obvious editing trace, thereby helping the user to easily shoot a personalized video with a personal style.
[0052] In the process of the video generation method of another embodiment of the present application, step 130 includes:
[0053] Sub-step A1: when the similarity between the first image feature information and the image feature information of any frame in the material image is greater than a preset similarity threshold, determining any frame in the material image as a material key frame.
[0054] For example, the material image is a video, which includes six frames, and the reference key frames have a total of five frames. First, based on the first reference key frame, the most similar frame is found in the six frames to determine a material key frame; then, based on the second reference key frame, the most similar frame is found in the six frames to determine a material key frame; and so on, five material key frames can be determined, and further, duplicate frames can be deleted.
[0055] In this process, the similarity between each reference key frame and all frames of the material image is calculated. The default is that the maximum similarity value is greater than the preset similarity threshold. Furthermore, it can be set that the preset similarity threshold cannot be lower than a certain value, such as not lower than "50%", that is, when the similarity between a reference key frame and all frames of the material image is less than "50%", the material key frame can be temporarily determined.
[0056] Optionally, image feature information of each frame in the material image is obtained, and the similarity of the image feature information between two frames is calculated using a related algorithm to obtain the similarity between the frames. Correspondingly, the image feature information of each frame in the material image is used to characterize the shooting content such as the shooting object and shooting environment in the corresponding frame.
[0057] In this embodiment, the similarity between the image feature information is used to determine the frame similar to the reference key frame in the material image as the key frame in the second video. It can be seen that the key frame in the second video is similar to the key frame in the first video, so the key frame in the second video can be processed in combination with some processing methods for the key frame in the first video, so that the second video can imitate the first video well.
[0058] In the process of the video generation method of another embodiment of the present application, after the step of acquiring the material key frame in the material image that matches the reference key frame, the method further includes the following steps:
[0059] Receives input for clip keyframes.
[0060] In some embodiments of the present application, the above input is used for the user to operate the material key frames automatically screened out by the large model to retain the material key frames after the operation. The input can be an operation. Exemplarily, the above input includes but is not limited to: the user uses a touch device such as a finger or a stylus to touch the controls on the screen, the touch input to the screen area, or a voice command input by the user, or a specific gesture input by the user, or other feasible inputs, which can be determined according to actual use requirements, and the embodiments of the present invention are not limited. The specific gesture in the embodiment of the present application can be any one of a single-click gesture, a sliding gesture, a drag gesture, a pressure recognition gesture, a long press gesture, an area change gesture, a double-press gesture, and a double-click gesture; the click input in the embodiment of the present application can be a single-click input, a double-click input, or any number of click inputs, etc., and can also be a long press input or a short press input.
[0061] In response to input of a material key frame, at least one of the following processes is performed:
[0062] The first item is, when the number of material key frames is at least two, adjusting the arrangement order of at least two material key frames.
[0063] like Figure 4 As shown, in the template details interface, the user clicks on the wonderful key frame editing option 4021, and a pop-up window 4022 is displayed, in which each material key frame is displayed in order, such as key frame 1, key frame 2, etc. For example, the above input can be that the user long presses a material key frame and drags it to another position to adjust the arrangement order between each material key frame.
[0064] The initial arrangement order of each material key frame is the order in which they appear in the material video from first to last; the initial arrangement order of each material key frame can also be the order in which the material images imported by the user are from first to last.
[0065] In this step, based on the user's adjustment of the arrangement order of the key frames of each material, they appear in the second video from first to last in the adjusted order.
[0066] Among them, in the second video, only the order of the key frames of each material is changed, and the order of non-key frames does not change.
[0067] For example, in the second video, the initial order is: frame 1, key frame 1, frame 2, key frame 2, and the adjusted order is: frame 1, key frame 2, frame 2, key frame 1.
[0068] The second item is to delete at least one material keyframe.
[0069] For example, the above input may be that the user long presses a key frame of a material to display a delete option, and the user clicks the delete option.
[0070] Among them, the deleted material key frames will still appear in the second video, but they will no longer be used as key frames.
[0071] The third item is to add at least one material keyframe.
[0072] For example, the above input can be that the user long presses a material key frame to display an add option, the user clicks the add option, all frames in the material image that have not been determined as material key frames are displayed, and the user clicks any frame to set the frame as a material key frame, and its position in the material key frame sequence is the one before the material key frame that the user long pressed.
[0073] For example, in the second video, the initial order is: frame 1, key frame 1, frame 2, key frame 2. Before the user adds frame 1 to key frame 2, the order changes to: key frame 1, frame 2, frame 1, key frame 2. In order to facilitate explanation that frame 1 is determined as a key frame, the order may also be: key frame 1, frame 2, key frame 3, key frame 2.
[0074] Optionally, the user replaces a certain material key frame through input. For example, the input may be that the user long presses a certain material key frame to display a replacement option, and the user clicks the replacement option to select a replacement frame.
[0075] It should be noted that the replacement here is equivalent to deletion followed by addition, so the replacement effect will not be described in detail.
[0076] In this embodiment, after the AI big model automatically selects the material key frames from the material image, the user can adjust the order between the selected material key frames so that the second video can be generated in the order adjusted by the user; the user can also delete the material key frames that he is not satisfied with or redundant; the user can also add the material key frames that he is satisfied with. It can be seen that in the second video, the material key frames confirmed by the user are used as the key frames of the video, so that more personalized needs of the user can be reflected in the second video, and more personalized details can be reflected in the second video.
[0077] In the process of the video generation method of another embodiment of the present application, step 140 includes:
[0078] Sub-step B1: Based on the first material key frame and the second image feature information in the material key frame, display the special effect identifier and related special effect identifiers of the first reference key frame matching the first material key frame, wherein a special effect identifier is used to indicate a special effect.
[0079] like Figure 4 As shown, in the template details interface, the user clicks on the keyframe special effect editing option 4031, and a pop-up window 4032 is displayed. The pop-up window 4032 displays the special effect identifiers of each material keyframe, such as keyframe 1 special effect, keyframe 2 special effect, etc.
[0080] Taking the first material key frame as an example, it is obtained based on the matching of the first reference key frame, and the special effects in the first reference key frame are extracted, so that the special effects mark of the first reference key frame is displayed in the pop-up window as the special effects mark of the first material key frame.
[0081] Furthermore, while displaying the special effect mark of the key frame of the material, a replacement frame special effect option is also displayed. When the user clicks the replacement frame special effect option, the replacement special effect mark is displayed as the related special effect mark. Figure 4 As shown on the outside of pop-up window 4032, through user input, the replaced special effect 1 logo to special effect 4 logo are displayed.
[0082] Optionally, the special effect logo is used to indicate a special effect, and the special effect logo displays the name of the special effect and text content describing the special effect.
[0083] For reference, special effects include magnification effects, reduction effects, and partial magnification effects.
[0084] Sub-step B2: receiving a second input of a first special effect identifier among the special effect identifier and related special effect identifiers.
[0085] In some embodiments of the present application, the second input is used for the user to select the first special effect identifier to trigger the addition of the first special effect indicated by the first special effect identifier in the first material key frame. The second input can be a second operation. Exemplarily, the second input includes but is not limited to: the user's touch input to the controls on the screen or the screen area through a touch device such as a finger or a stylus, or a voice command input by the user, or a specific gesture input by the user, or other feasible inputs, which can be determined according to actual use requirements and are not limited in the embodiments of the present invention. The specific gesture in the embodiment of the present application can be any one of a single-click gesture, a sliding gesture, a drag gesture, a pressure recognition gesture, a long press gesture, an area change gesture, a double-press gesture, and a double-click gesture; the click input in the embodiment of the present application can be a single-click input, a double-click input, or any number of click inputs, etc., and can also be a long press input or a short press input.
[0086] For example, the second input may be Figure 4 In the pop-up window 4032 shown, the user clicks on the key frame 1 special effect logo; or, triggered by user input, the special effect 1 logo, special effect 2 logo, special effect 3 logo, and special effect 4 logo are displayed, and the user clicks on one of the logos.
[0087] Sub-step B3: In response to the second input, applying the first special effect indicated by the first special effect identifier to the first material key frame.
[0088] For example, in the first reference key frame, the face of the character is magnified, that is, a local magnification special effect is applied, so that in the first material key frame, the face of the character is also magnified.
[0089] In this embodiment, in the material key frame, the special effects in the reference key frame matching it are applied, so that the special effects of the material key frame are similar to the special effects of the reference key frame, so that the special effects of the generated second video can be similar to the special effects of the first video. In addition, the user can also replace the special effects to make the second video more personal.
[0090] In the process of the video generation method of another embodiment of the present application, step 140 includes:
[0091] Sub-step C1: Displaying the identifier of the sound-emitting object in the first video and the text content converted from the audio content emitted by the sound-emitting object, wherein one identifier is used to indicate one sound-emitting object.
[0092] like Figure 4As shown, in the template details interface, the user clicks on the sound source dubbing text editing option 4041, and a pop-up window 4042 is displayed. The pop-up window 4042 displays the identifiers of each sound object in the first video, such as thumbnails of each sound source in the first video; correspondingly, next to the identifier of the sound object, text content converted from the audio content emitted by the sound object is displayed.
[0093] Optionally, the AI big model obtains the audio content in the first video, and classifies the audio content with the same sound characteristics together, and then finds the corresponding sound source as the sound object.
[0094] Sub-step C2: receiving a third input of the first text content of the first sound object among the sound objects.
[0095] In some embodiments of the present application, the third input is used by the user to edit the first text content of the first sound object, and correspond the first sound object to the second sound object in the material image. The third input can be a third operation. Exemplarily, the third input includes but is not limited to: the user uses a touch device such as a finger or a stylus to touch the controls on the screen, the touch input to the screen area, or a voice command input by the user, or a specific gesture input by the user, or other feasible inputs, which can be determined according to actual use requirements, and the embodiments of the present invention are not limited. The specific gesture in the embodiment of the present application can be any one of a single-click gesture, a sliding gesture, a drag gesture, a pressure recognition gesture, a long press gesture, an area change gesture, a double-press gesture, and a double-click gesture; the click input in the embodiment of the present application can be a single-click input, a double-click input, or any number of click inputs, etc., and can also be a long press input or a short press input.
[0096] For example, the third input may be that the user double-clicks the first text content in the pop-up window to put the first text content into an editing state, and after the user edits the first text content, the second text content is obtained.
[0097] Wherein, the second sound object is associated with the third input.
[0098] For example, the third input also includes that the user long presses the identifier of the first sound-emitting object, thereby displaying a list of all object identifiers that can emit sound in the material image, and the user clicks on one of the object identifiers to use the object identifier as the second sound-emitting object.
[0099] Sub-step C3: In response to the third input, the first text content is replaced with the second text content associated with the third input, and in the process of generating the second video, the second text content is converted into audio content emitted by the second sound object in the material image.
[0100] In this step, in the process of generating the second video, the second text content is converted into audio content, and the audio content is emitted by the second sound object.
[0101] Optionally, the second sound object makes a sound in a similar frame in the second video with reference to the frame corresponding to the sound made by the first sound object. For example, in the first video, a person sits on a bench and admires the beauty of the sunset; thus, in the second video, the user sits on a bench alone and can also make the same exclamation.
[0102] In this embodiment, the audio content of a sound-making object in the first video is copied to the sound-making object shot by the user, so that the video shot by the user can replicate the audio environment of character dialogues, character emotions, animal calls, telephone ringtones, etc. in the first video; in addition, the user can also fine-tune the audio content based on the actual shooting environment, so that the user can not only refer to the first video to shoot the video, but also retain his own shooting characteristics in the shot video, thereby making the final generated video personalized and authentic.
[0103] Combined with the previous embodiment, Figure 4 The pop-up window 4042 shown may also display the identification of the sound-emitting object in the material image that has not been replaced by the sound-emitting object in the first video, and text content converted from the audio content emitted by the corresponding sound-emitting object. Therefore, the user may also modify the text content here to slightly adjust the original sound while retaining the original sound shot by the user in the second video.
[0104] In the process of the video generation method of another embodiment of the present application, step 140 further includes:
[0105] Sub-step D1: Display the sound type and related sound types of the sound emitted by the first sound-emitting object.
[0106] like Figure 4 As shown, in the template details interface, the user clicks on the sound type editing option 4051, and a pop-up window 4052 is displayed. The pop-up window 4052 displays the identification of each sound-emitting object, such as the thumbnail icon of the sound source appearing in the first video; at the same time, the sound type emitted by each sound-emitting object in the first video is displayed.
[0107] Among them, the timbre of a character itself is regarded as a voice type. More interesting voice types include: girls, boys, children, dialects, etc.
[0108] like Figure 4As shown, in the pop-up window 4052, the default display is the own timbre of each sound-making object in the first video as its sound type; a sound replacement option is also displayed. After the user clicks on the sound replacement option, type 1, type 2, etc. are displayed as related sound types.
[0109] Wherein, based on the aforementioned embodiment, if the second sound-emitting object in the material image can emit the audio content of the first sound-emitting object in the first video, the sound type of the second sound-emitting object is used as a replacement sound type.
[0110] Furthermore, for example, if a person and a dog appear in the first video, and a person and a dog also appear in the material image, when displaying the sound type of the person in the first video, the model automatically matches the person in the first video and the person in the material image, thereby using the sound type of the person in the material image as a replacement sound type. Similarly, when displaying the sound type of the dog in the first video, the model automatically matches the dog in the first video and the dog in the material image, thereby using the sound type of the dog in the material image as a replacement sound type.
[0111] Sub-step D2: receiving a fourth input of a sound type and a first sound type among related sound types.
[0112] In some embodiments of the present application, the fourth input is used by the user to select the first sound type of the first sound object, so that the sound emitted by the first sound object has the sound characteristic information of the first sound type. The fourth input can be a fourth operation. Exemplarily, the fourth input includes but is not limited to: the user's touch input to the controls on the screen or the screen area through a touch device such as a finger or a stylus, or a voice command input by the user, or a specific gesture input by the user, or other feasible inputs, which can be determined according to actual use requirements and are not limited in the embodiments of the present invention. The specific gesture in the embodiment of the present application can be any one of a single-click gesture, a sliding gesture, a drag gesture, a pressure recognition gesture, a long press gesture, an area change gesture, a double-press gesture, and a double-click gesture; the click input in the embodiment of the present application can be a single-click input, a double-click input, or any number of click inputs, etc., and can also be a long press input or a short press input.
[0113] For example, in combination with the user's pre-operation to make the second sound object in the material image emit the audio content of the first sound object, the fourth input may be that the user selects the sound type of the second sound object's own timbre as the first sound type in the replacement sound type. In which, along with the display of the replacement sound type, an audition mark is also displayed. After the user clicks the audition mark, the audition audio of the sound type is played, and the user can make a selection after the audition.
[0114] Sub-step D3: In response to the fourth input, in the process of generating the second video, the sound feature information of the first sound type is used as the sound feature information of the sound emitted by the second sound-emitting object.
[0115] In this step, in the process of generating the second video, at least the following effect can be achieved: first, referring to the aforementioned embodiment, character A filmed by the user himself is allowed to say what character B in the first video said; and then in this embodiment, it is set to retain character A's own voice in the second video.
[0116] In this embodiment, on the basis of utilizing the audio content in the first video, the sound characteristics of the object photographed by the user himself can also be retained, so that the generated second video not only reproduces the audio environment such as the character dialogue in the first video, but also has no obvious traces of reproduction, making the second video more realistic.
[0117] Combined with the previous embodiment, Figure 4 The pop-up window 4052 shown may also display the identification of the sound-emitting object in the material image that has not been replaced by the sound-emitting object in the first video, as well as the sound type and related sound types of the corresponding sound-emitting object. Therefore, the user may also confirm the sound type here to retain the original sound features shot by the user in the second video, or add some interesting sound changes.
[0118] In the process of the video generation method of another embodiment of the present application, step 140 includes the following steps:
[0119] The video style identifier of the first video and related video style identifiers are displayed, wherein a video style identifier is used to indicate a video style.
[0120] like Figure 4 As shown, in the template details interface, the user clicks on the video style editing option 4061, and a pop-up window 4062 is displayed. The pop-up window 4062 displays the video style identifier of the first video and the replacement style option. After the user clicks on the replacement style option, the style 1 identifier, the style 2 identifier, and the style 3 identifier are displayed. Among them, the style identifier displayed here under the user's trigger is the style identifier of the related video.
[0121] A fifth input of a first video style identifier among the video style identifiers and the related video style identifiers is received.
[0122] In some embodiments of the present application, the fifth input is used for the user to select the first video style identifier so that the video style of the second video is the first video style indicated by the first video style identifier. The fifth input can be a fifth operation. Exemplarily, the fifth input includes but is not limited to: the user's touch input to the controls on the screen or the screen area through a touch device such as a finger or a stylus, or a voice command input by the user, or a specific gesture input by the user, or other feasible inputs, which can be determined according to actual use requirements and are not limited in the embodiments of the present invention. The specific gesture in the embodiment of the present application can be any one of a single-click gesture, a sliding gesture, a drag gesture, a pressure recognition gesture, a long press gesture, an area change gesture, a double-press gesture, and a double-click gesture; the click input in the embodiment of the present application can be a single-click input, a double-click input, or any number of click inputs, etc., and can also be a long press input or a short press input.
[0123] For example, the fifth input may be that the user clicks on any video style identifier.
[0124] In response to the fifth input, in the process of generating the second video, the first video style indicated by the first video style identifier is applied to the second video.
[0125] The application method includes, but is not limited to: adjusting the image tone of the second video, and adjusting the scene reference object in the second video. The scene reference object may include a photographed object in the background image.
[0126] For example, if the first video style is the video style of the first video, the overall tone of the material image can be adjusted with reference to the overall tone of the first video; if the first video style is the video style of the first video, such as the rural style, the scene reference objects in the material image can be replaced with reference to the scene reference objects such as buildings and crops in the first video, etc.
[0127] In this embodiment, a method for applying a video style is provided, so that the second video can restore the video style in the first video and can also reflect the user's personalized video style, thereby more conveniently meeting the user's needs for shooting personalized videos.
[0128] In the process of the video generation method of another embodiment of the present application, step 140 includes the following steps:
[0129] Display the background sound identifier of the clip in the first video and the related background sound identifiers, wherein a background sound identifier is used to indicate a background sound.
[0130] Optionally, a background sound such as a background music.
[0131] like Figure 4As shown, in the template details interface, the user clicks on the background sound edit option 4071, and a pop-up window 4072 is displayed, in which the background sound identifiers of each clip in the first video are displayed. Each clip also displays a replace background sound option, and after the user clicks on the background sound option, the sound 1 identifier and the sound 2 identifier are displayed. Among them, the sound identifier displayed here under the user's trigger is the relevant background sound identifier.
[0132] Optionally, multiple consecutive frames with similar image features are regarded as a segment.
[0133] Optionally, the background sound identifier displays the name of the background sound.
[0134] A sixth input of a background sound identifier of a first segment in a first video and a first background sound identifier among the related background sound identifiers is received.
[0135] In some embodiments of the present application, the sixth input is used for the user to select the first background sound identifier for the first segment to trigger the first background sound indicated by the first background sound identifier to be played in the second segment of the second video. The sixth input can be a sixth operation. Exemplarily, the sixth input includes but is not limited to: the user's touch input to the controls on the screen or the screen area through a touch device such as a finger or a stylus, or a voice command input by the user, or a specific gesture input by the user, or other feasible inputs, which can be determined according to actual use requirements and are not limited in the embodiments of the present invention. The specific gesture in the embodiment of the present application can be any one of a single-click gesture, a sliding gesture, a drag gesture, a pressure recognition gesture, a long press gesture, an area change gesture, a double-press gesture, and a double-click gesture; the click input in the embodiment of the present application can be a single-click input, a double-click input, or any number of click inputs, etc., and can also be a long press input or a short press input.
[0136] For example, the sixth input may be that the user selects a background sound identifier from the background sound identifiers of the first segment and the related sound identifiers as the first background sound identifier.
[0137] Optionally, an audition mark is displayed along with the background sound identifier, and the user clicks the audition mark to play the corresponding audition sound.
[0138] Optionally, in the process of generating the second video, several consecutive frames with similar image feature information are taken as a segment.
[0139] When the similarity between the image features of the first segment and the image features of a segment in the video to be generated is greater than a preset similarity threshold, the first segment is assumed to be similar to the segment, that is, the segment is determined as the second segment, so that the background sound of the first segment is added to the second segment.
[0140] Optionally, a set of image features of the first segment is obtained, represented as a first vector; a set of image features of each segment in the video to be generated is obtained, represented as multiple second vectors; when the similarity score between the first vector and one of the second vectors is greater than a preset similarity threshold, the segment corresponding to the second vector is taken as the second segment.
[0141] In response to the sixth input, in the process of generating the second video, the first background sound indicated by the first background sound identifier is added to the second segment.
[0142] In this embodiment, the background sound of a similar segment in the first video is restored in a segment of the second video, so that the atmosphere brought by the background sound of the first video can be created in the second video, without the user having to search through a large amount of background sounds such as music, thereby making it easier for the user to shoot personalized videos. In addition, the user can also change the background sound to diversify the video effect, and will not be constrained by the first video while using the first video.
[0143] In a video generation method according to another embodiment of the present application, Figure 4 As shown, after the user completes editing of each option in the template details interface, he clicks the generate option 408, thereby playing the generated second video in the video preview window 401. Further, the user can continue editing until the user is satisfied with the currently generated video through previewing the video.
[0144] In another embodiment of the video generation method of the present application, after the user imports the material image, he can directly click the generate option to play the generated initial video in the video preview window. In the initial video, the AI big model automatically selects the material keyframes, automatically applies special effects to the material keyframes, automatically matches the sound objects in the material image with the sound objects in the first video, automatically retains the sound features shot by the user, automatically retains the video style of the first video, and automatically adds the background sound of the first video. Furthermore, based on the generated initial video, the user can Figure 4 The template details interface is shown in the figure, and the user can edit the template until the user is satisfied with the video to be played.
[0145] In the video generation method of another embodiment of the present application, the user can click the save option to save the second video in the album; the user can also click the share option to share the second video with friends.
[0146] In more embodiments, the generated second video can be used as a video template to be applied in video production.
[0147] In summary, in this application, users screen out high-quality videos, and the AI big model uses the high-quality videos as video templates. When users shoot videos or photos and other materials, the AI big model can generate videos similar to the video template based on the video template and the materials. Among them, the AI big model intelligently identifies the reference key frames, reference key frame special effects, sound objects, audio, background sounds, video styles, etc. in the video template, and displays them as template content in the editing interface. Users can make relevant edits so that the video template can be more personalized and applied to the materials shot by the user, thereby generating their ideal videos.
[0148] The video generation method provided in the embodiment of the present application can be executed by a video generation device. In the embodiment of the present application, the video generation device provided in the embodiment of the present application is described by taking the video generation method executed by the video generation device as an example.
[0149] Figure 5 A block diagram of a video generating device according to an embodiment of the present application is shown, the device comprising:
[0150] A first acquisition module 10 is used to acquire first image feature information of a reference key frame in a first video;
[0151] The receiving module 20 is used to receive a first input of a material image;
[0152] A second acquisition module 30 is used to acquire a material key frame matching the reference key frame in the material image according to the first image feature information in response to the first input;
[0153] The generation module 40 is used to generate a second video according to the second image feature information, the material key frame and the material image; wherein the second image feature information at least includes: special effect information of the first video, sound information of the first video, and style information of the first video.
[0154] In an embodiment of the present application, first, the first image feature information of the reference key frame in the first video is obtained. The first video can be a related video selected by the user based on the current shooting environment. Here, the first video can be used as a video template. Then, the user uploads the material image, which can be an image shot by the user in the current shooting environment, so as to obtain the material key frame matching the reference key frame in the material image by using the first image feature information of the reference key frame of the first video; further, the second image feature information of the first video is referred to, and the extracted material key frame and the material image itself are combined to generate the second video. Among them, the second image feature information is not limited to the special effect information, sound information and style information of the first video. It can be seen that in an embodiment of the present application, in the shooting scene, the user can select a high-quality video such as a video template on the Internet, so as to integrate some shooting effects, the environmental atmosphere presented, etc. in the video template into the video shot by the user in combination with artificial intelligence technology, so that some video effects are added to the shot video, but there is no obvious editing trace, thereby helping the user to easily shoot a personalized video with a personal style.
[0155] Optionally, the second acquisition module 30 includes:
[0156] The determination unit is configured to determine any frame in the material image as a material key frame when the similarity between the first image feature information and the image feature information of any frame in the material image is greater than a preset similarity threshold.
[0157] Optionally, the generating module 40 includes:
[0158] A first display unit is used to display a special effect identifier and related special effect identifiers of a first reference key frame matching the first material key frame according to the first material key frame and the second image feature information in the material key frame; wherein one special effect identifier is used to indicate one special effect;
[0159] A first receiving unit, configured to receive a second input of a first special effect identifier among the special effect identifier and the related special effect identifiers;
[0160] The application unit is configured to apply, in response to a second input, a first special effect indicated by a first special effect identifier to a first material key frame.
[0161] Optionally, the generating module 40 includes:
[0162] A second display unit is used to display the identifier of the sound-emitting object in the first video and the text content converted from the audio content emitted by the sound-emitting object; wherein one identifier is used to indicate one sound-emitting object;
[0163] A second receiving unit is used to receive a third input of a first text content of a first sound object among the sound objects;
[0164] A first processing unit is configured to replace the first text content with second text content associated with the third input in response to the third input, and convert the second text content into audio content emitted by a second sound object in the material image during the process of generating the second video;
[0165] Wherein, the second sound object is associated with the third input.
[0166] Optionally, the generating module 40 further includes:
[0167] A third display unit, used to display the sound type and related sound types of the sound emitted by the first sound emitting object;
[0168] A third receiving unit, configured to receive a fourth input of a first sound type among the sound type and the related sound types;
[0169] The second processing unit is used to respond to the fourth input and use the sound feature information of the first sound type as the sound feature information of the sound emitted by the second sound-emitting object in the process of generating the second video.
[0170] The device in the application embodiment can be an electronic device or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc. It can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., and the present application embodiment is not specifically limited.
[0171] The device of the embodiment of the present application may be a device having an action system. The action system may be an Android action system, an iOS action system, or other possible action systems, which are not specifically limited in the embodiment of the present application.
[0172] The device provided in the embodiment of the present application can implement each process implemented in the above method embodiment and achieve the same technical effect. To avoid repetition, it will not be repeated here.
[0173] Alternatively, if Figure 6 As shown, an embodiment of the present application also provides an electronic device 100, including a processor 101, a memory 102, and a program or instruction stored in the memory 102 and executable on the processor 101. When the program or instruction is executed by the processor 101, each step of any of the above-mentioned video generation method embodiments is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
[0174] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.
[0175] Figure 7 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.
[0176] The electronic device 1000 includes but is not limited to: a radio frequency unit 1001, a network module 1002, an audio output unit 1003, an input unit 1004, a sensor 1005, a display unit 1006, a user input unit 1007, an interface unit 1008, a memory 1009, a processor 1010, a camera 1011 and other components.
[0177] Those skilled in the art will appreciate that the electronic device 1000 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 1010 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Figure 7 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.
[0178] Among them, the processor 1010 is used to obtain the first image feature information of the reference key frame in the first video; the user input unit 1007 is used to receive a first input of the material image; the processor 1010 is also used to respond to the first input and, according to the first image feature information, obtain the material key frame in the material image that matches the reference key frame; generate a second video according to the second image feature information, the material key frame and the material image; wherein the second image feature information includes at least: the special effects information of the first video, the sound information of the first video, and the style information of the first video.
[0179] In an embodiment of the present application, first, the first image feature information of the reference key frame in the first video is obtained. The first video can be a related video selected by the user based on the current shooting environment. Here, the first video can be used as a video template. Then, the user uploads the material image, which can be an image shot by the user in the current shooting environment, so as to obtain the material key frame matching the reference key frame in the material image by using the first image feature information of the reference key frame of the first video; further, the second image feature information of the first video is referred to, and the extracted material key frame and the material image itself are combined to generate the second video. Among them, the second image feature information is not limited to the special effect information, sound information and style information of the first video. It can be seen that in an embodiment of the present application, in the shooting scene, the user can select a high-quality video such as a video template on the Internet, so as to integrate some shooting effects, the environmental atmosphere presented, etc. in the video template into the video shot by the user in combination with artificial intelligence technology, so that some video effects are added to the shot video, but there is no obvious editing trace, thereby helping the user to easily shoot a personalized video with a personal style.
[0180] Optionally, the processor 1010 is further configured to determine any frame in the material image as a material key frame when the similarity between the first image feature information and image feature information of any frame in the material image is greater than a preset similarity threshold.
[0181] Optionally, the processor 1010 is also used to display the special effect identifier and related special effect identifiers of the first reference key frame matching the first material key frame according to the first material key frame and the second image feature information; wherein one of the special effect identifiers is used to indicate a special effect; the user input unit 1007 is also used to receive a second input of a first special effect identifier among the special effect identifiers and the related special effect identifiers; the processor 1010 is also used to apply the first special effect indicated by the first special effect identifier to the first material key frame in response to the second input.
[0182] Optionally, the display unit 1006 is used to display the identifier of the sound-making object in the first video, and the text content converted from the audio content emitted by the sound-making object; wherein one of the identifiers is used to indicate one of the sound-making objects; the user input unit 1007 is also used to receive a third input of the first text content of the first sound-making object among the sound-making objects; the processor 1010 is also used to replace the first text content with the second text content associated with the third input in response to the third input, and in the process of generating the second video, convert the second text content into the audio content emitted by the second sound-making object in the material image; wherein the second sound-making object is associated with the third input.
[0183] Optionally, the display unit 1006 is further used to display the sound type and related sound types of the sound emitted by the first sound-emitting object; the user input unit 1007 is further used to receive a fourth input of a first sound type among the sound types and the related sound types; the processor 1010 is further used to respond to the fourth input and, in the process of generating the second video, use the sound characteristic information of the first sound type as the sound characteristic information of the sound emitted by the second sound-emitting object.
[0184] In summary, in this application, users screen out high-quality videos, and the AI big model uses the high-quality videos as video templates. When users shoot videos or photos and other materials, the AI big model can generate videos similar to the video template based on the video template and the materials. Among them, the AI big model intelligently identifies the reference key frames, reference key frame special effects, sound objects, audio, background sounds, video styles, etc. in the video template, and displays them as template content in the editing interface. Users can make relevant edits so that the video template can be more personalized and applied to the materials shot by the user, thereby generating their ideal videos.
[0185] It should be understood that in the embodiment of the present application, the input unit 1004 may include a graphics processor (Graphics Processing Unit, GPU) 10041 and a microphone 10042, and the graphics processor 10041 processes the image data of the static picture or video image obtained by the image capture device (such as a camera) in the video image capture mode or the image capture mode. The display unit 1006 may include a display panel 10061, and the display panel 10061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1007 includes a touch panel 10071 and at least one of other input devices 10072. The touch panel 10071 is also called a touch screen. The touch panel 10071 may include two parts: a touch detection device and a touch controller. Other input devices 10072 may include, but are not limited to, a physical keyboard, a function key (such as a volume control button, a switch button, etc.), a trackball, a mouse, and an action rod, which will not be repeated here. The memory 1009 can be used to store software programs and various data, including but not limited to applications and action systems. The processor 1010 may integrate an application processor and a modem processor, wherein the application processor mainly processes the action system, user pages and application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the modem processor may not be integrated into the processor 1010.
[0186] The memory 1009 can be used to store software programs and various data. The memory 1009 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 1009 may include a volatile memory or a non-volatile memory, or the memory 1009 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 1009 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.
[0187] The processor 1010 may include one or more processing units; optionally, the processor 1010 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 1010.
[0188] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned video generation method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
[0189] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.
[0190] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned video generation method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.
[0191] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.
[0192] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned video generation method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.
[0193] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.
[0194] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of each embodiment of the present application.
[0195] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.
Claims
1. A video generation method, characterized in that: The method comprises: Acquire first image feature information of a reference key frame in a first video; Receiving a first input of a material image; In response to the first input, acquiring a material key frame in the material image that matches the reference key frame according to the first image feature information; A second video is generated according to the second image feature information, the material key frame and the material image; wherein the second image feature information includes at least: special effect information of the first video, sound information of the first video, and style information of the first video.
2. The method according to claim 1, characterized in that The acquiring, according to the first image feature information, a material key frame in the material image that matches the reference key frame includes: When the similarity between the first image feature information and the image feature information of any frame in the material image is greater than a preset similarity threshold, any frame in the material image is determined as a material key frame.
3. The method according to claim 1, characterized in that The step of generating a second video according to the second image feature information, the material key frame and the material image includes: According to the first material key frame in the material key frame and the second image feature information, displaying the special effect identifier of the first reference key frame matching the first material key frame and the related special effect identifiers; wherein one of the special effect identifiers is used to indicate a special effect; Receiving a second input of a first special effect identifier among the special effect identifier and related special effect identifiers; In response to the second input, a first special effect indicated by the first special effect identifier is applied to the first material key frame.
4. The method according to claim 1, characterized in that The step of generating a second video according to the second image feature information, the material key frame and the material image includes: Displaying the identifier of the sound-making object in the first video and text content converted from the audio content emitted by the sound-making object; wherein one identifier is used to indicate one sound-making object; receiving a third input of a first text content of a first sound object among the sound objects; In response to the third input, the first text content is replaced with second text content associated with the third input, and in the process of generating the second video, the second text content is converted into audio content emitted by a second sound object in the material image; Wherein, the second sound object is associated with the third input.
5. The method according to claim 4, characterized in that The step of generating a second video according to the second image feature information, the material key frame and the material image further includes: Displaying the sound type and related sound types of the sound emitted by the first sound-emitting object; receiving a fourth input of a first sound type among the sound type and the related sound types; In response to the fourth input, in the process of generating the second video, the sound feature information of the first sound type is used as the sound feature information of the sound emitted by the second sound-emitting object.
6. A video generating device, characterized in that: The device comprises: A first acquisition module, used to acquire first image feature information of a reference key frame in a first video; A receiving module, used for receiving a first input of a material image; A second acquisition module, configured to acquire, in response to the first input, a material key frame in the material image that matches the reference key frame according to the first image feature information; A generation module is used to generate a second video based on second image feature information, the material key frame and the material image; wherein the second image feature information at least includes: special effect information of the first video, sound information of the first video, and style information of the first video.
7. The device according to claim 6, characterized in that The second acquisition module includes: The determining unit is configured to determine any frame in the material image as a material key frame when the similarity between the first image feature information and the image feature information of any frame in the material image is greater than a preset similarity threshold.
8. The device according to claim 6, characterized in that The generating module comprises: A first display unit, configured to display a special effect identifier and related special effect identifiers of a first reference key frame matching the first material key frame according to the first material key frame and the second image feature information; wherein one special effect identifier is used to indicate a special effect; A first receiving unit, configured to receive a second input of a first special effect identifier among the special effect identifier and related special effect identifiers; An application unit is configured to apply, in response to the second input, a first special effect indicated by the first special effect identifier to the first material key frame.
9. The device according to claim 6, characterized in that The generating module comprises: A second display unit is used to display the identifier of the sound-emitting object in the first video and the text content converted from the audio content emitted by the sound-emitting object; wherein one identifier is used to indicate one sound-emitting object; A second receiving unit, configured to receive a third input of a first text content of a first sound object among the sound objects; a first processing unit, configured to replace the first text content with second text content associated with the third input in response to the third input, and convert the second text content into audio content emitted by a second sound-emitting object in the material image during the process of generating the second video; Wherein, the second sound object is associated with the third input.
10. The device according to claim 9, characterized in that The generating module further includes: a third display unit, configured to display a sound type and related sound types of the sound emitted by the first sound emitting object; A third receiving unit, configured to receive a fourth input of the first sound type among the sound type and the related sound types; The second processing unit is used to respond to the fourth input and use the sound feature information of the first sound type as the sound feature information of the sound emitted by the second sound-emitting object in the process of generating the second video.
