Video synthesis method and device, electronic equipment and storage medium

Abstract

The invention discloses a video synthesis method and device, electronic equipment and a storage medium, and the method comprises the steps: collecting an image and audio uploaded by a user, extracting an image background, character features and audio features, generating a target story script according to the extracted image background, character features and audio features, and carrying out the video synthesis according to the target story script. And generating a target video according to the target story script. An image and an audio of a user are acquired, audio features, a picture background and character features of the user are extracted, a story script is generated according to the audio features, the picture background and the character features of the user, and a target video only belonging to the user is generated. Original static photos are automatically changed into dynamic videos through processing of an artificial intelligence technology, so that the interestingness of the photos is increased, and the interactivity between the photo album and the user is enhanced.

Description

Technical Field

[0001] The present invention relates to the field of video synthesis, and in particular to a video synthesis method, device, electronic equipment and storage medium. Background Art

[0002] With the popularity of mobile devices, existing mobile devices generally have the function of taking pictures and storing and displaying the taken images. However, with the rapid development of computer vision technology, people are paying more and more attention to image processing technology and audio processing technology. Most picture-to-video synthesis requires users to manually synthesize and add background music. The synthesis process is relatively complicated, which makes the interaction between the albums of existing mobile devices and users not strong. Therefore, a method that can automatically synthesize videos is urgently needed to increase the interactivity with users. Summary of the invention

[0003] The embodiments of the present invention provide a video synthesis method, device, electronic device and storage medium to automatically synthesize pictures in a user's photo album into a video and increase the interactivity with the user.

[0004] The first aspect of the present application provides a video synthesis method, comprising: Collect images and audio uploaded by users; Extract image background, character features and audio features; Generate a target story script according to the extracted image background, character features, and audio features; Generate a target video based on the target story script.

[0005] Optionally, the collecting of audio uploaded by the user may then include: According to the audio uploaded by the user, the user's voice frequency and voice timbre are obtained, and the user's voice frequency and voice timbre are automatically saved.

[0006] Optionally, generating a target story script according to the extracted image background, character features, and audio features includes: Analyzing the image background using artificial intelligence technology to generate a story background based on the image background; Generate a dynamic character and target audio according to the extracted character features and audio features; A target story script is generated based on the acquired story background, the dynamic characters and the target audio.

[0007] Optionally, generating a target video according to a target story script includes: Generate a preliminary video according to the target story script; Background audio is added to the preliminary video to generate a target video.

[0008] Optionally, the target audio is obtained by using audio synthesis technology.

[0009] Optionally, the background audio is generated according to the target story script, including: Obtain the story background, dynamic characters and target audio generated by the target story script; Generate appropriate background audio based on the story background, dynamic characters and target audio generated by the story script.

[0010] Optionally, generating a target video according to a target story script, then comprising: If you are not satisfied with the generated target video, you can edit and modify the generated target video again through a voice intelligent assistant or manual operation, and back up and store the modified target video.

[0011] A second aspect of the present application provides a video synthesis device, including: The acquisition module is used to collect images and audio uploaded by users; An extraction module, used to extract image background, character features and audio features; A processing module, used for generating a target story script according to the extracted image background, character features and audio features; The video generation module is used to generate a target video according to a target story script.

[0012] A third aspect of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned video synthesis method when executing the computer program.

[0013] A fourth aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned video synthesis method is implemented.

[0014] The above-mentioned video synthesis method, device, electronic device and storage medium collect images and audio uploaded by users, extract image background, character features and audio features, generate a target story script based on the extracted image background, character features and audio features, and generate a target video based on the target story script. By acquiring the user's images and audio, extracting the user's audio features and picture background and character features, generating a story script based on the user's audio features and picture background and character features, generating a target video that belongs only to the user, and automatically transforming the originally static photos into dynamic videos through the processing of artificial intelligence technology, which increases the fun of the photos and enhances the interactivity between the album and the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0016] Figure 1 is a flow chart of a video synthesis method in one embodiment of the present invention; Figure 2 is a schematic diagram of a video synthesis device according to an embodiment of the present invention; Figure 3 is a schematic diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] The video synthesis method provided by the embodiment of the present invention can be applied as follows: Figure 1 Specifically, the video synthesis method is applied in a video synthesis system, and the video synthesis system includes: Figure 1 The client and server shown in the figure communicate with each other through the network to realize the automatic transformation of static photos into dynamic videos through artificial intelligence technology. The client, also known as the user end, refers to the program corresponding to the server that provides local services to customers. The client can be installed on, but not limited to, various personal computers, laptops, smart phones, tablets, and portable wearable devices. The server can be implemented as an independent server or a server cluster consisting of multiple servers.

[0019] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0020] It should also be understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0021] As used in the present specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]", depending on the context.

[0022] In addition, in the description of the present specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0023] References to "one embodiment" or "some embodiments" etc. described in the present specification mean that one or more embodiments of the present invention include specific features, structures or characteristics described in conjunction with the embodiment. Thus, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. appearing at different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0024] It should be understood that the order of execution of the steps in the following embodiments does not imply a precedence of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0025] In order to illustrate the technical solution of the present invention, specific embodiments are provided below for illustration.

[0026] In one embodiment, if Figure 1 As shown, a video synthesis method is provided, which is applied in Figure 1 The following steps are used as an example: S10: Collect images and audio uploaded by users.

[0027] S20: Extracting image background, character features and audio features.

[0028] Specifically, images and audio uploaded by the user are collected. The uploaded images can be pictures taken by the user through a mobile terminal, can be user pictures stored in the mobile terminal device, or can be pictures obtained by other methods that can obtain user photos. The specific method of obtaining the photos is not specifically limited. The uploaded user audio is obtained by collecting the user audio using an audio collection device. The audio collection device can be an audio collection device that comes with the mobile device, or an external audio collection device. The specific method of collecting audio is also not limited. After obtaining the images and audio uploaded by the user, it is necessary to process the images uploaded by the user, divide the images uploaded by the user into image background and character parts, and extract features from each part to ensure that the script is designed according to the images uploaded by the user.

[0029] After the user has collected audio, the user's audio features are extracted based on the audio uploaded by the user. The audio features include obtaining the user's audio timbre, audio frequency and other audio features when the user speaks. The user's audio features are extracted to facilitate adding the user audio to the synthesized video.

[0030] In this embodiment, by collecting images and audio uploaded by users and extracting image background, character features and audio features, the background, character features in the images uploaded by users and the audio features of the user are obtained, and a unique video of the user is synthesized based on the image background, character features and audio features, thereby increasing the uniqueness and interest of the user's video.

[0031] S30: Generate a target story script according to the extracted image background, character features and audio features.

[0032] Specifically, after extracting the background, character features and audio features of the image uploaded by the user, a target story script is generated based on the image background, the character features and the audio features.

[0033] In one embodiment, generating a target story script according to the extracted image background, character features, and audio features specifically includes the following steps: S31: Analyze the image background using artificial intelligence technology to generate a story background according to the image background.

[0034] S32: Generate a dynamic character and target audio according to the extracted character features and audio features.

[0035] S33: Generate a target story script based on the acquired story background, the dynamic characters and the target audio.

[0036] Specifically, artificial intelligence technology is used to analyze the background of the image, and the story background is generated based on the background of the image. For example, if the background of the image is a river, the story background generated based on the background of the picture is a gurgling river. If the background of the image is trees, the story background generated is leaves fluttering in the wind.

[0037] After extracting the image background, the character features in the image will also be extracted. The extracted characters will be generated into dynamic character movements according to the current behavior of the characters in the image or the direction of movement. For example, if the extracted character features are running, the character movement will be added according to the direction of movement in the image. If the extracted character features are dancing, dance movements will be added according to the direction of the character dance, so that the static characters in the photo become dynamic, increasing the interest of the photo.

[0038] The extraction of user audio is mainly done by having the user record his or her own speaking audio in a quiet environment, uploading the audio of his or her own speaking, and extracting audio features from the uploaded user audio, such as the audio frequency, the user's vocalization habits and other user audio timbre features. For the audio with extracted audio features, the target audio with the user's speaking audio features can be generated through speech synthesis technology. Based on the user audio, the story script will be designed and the final target story script will be generated.

[0039] Optionally, in one embodiment, the background audio is generated according to the target story script. For example, if the target story script is by the river, the sound of the river will be added accordingly. If there are trees in the target story script, the sound of wind blowing leaves can be added. If there are birds in the target story script, the sound of birdsong can be added. According to the script of the target story, the corresponding audio is added to the target video to make the synthesized target video more reasonable. The examples in this embodiment are only for illustration and are not limited in any way.

[0040] In this embodiment, the image background is analyzed by using artificial intelligence technology, a story background is generated according to the image background, dynamic characters and target audio are generated according to the extracted character features and audio features, and a target story script is generated according to the acquired story background, the dynamic characters and the target audio. This application generates a story background based on the picture background to increase the richness of the video background, adds dynamic characters based on the character features in the image, adds the character's actions based on the character features, increases the user's interest, and generates target audio based on the user audio, adds audio with user audio as the narration in the appropriate place of the story script, increases the richness of the story script, and makes static photos dynamic by generating a story background, dynamic characters in the story and adding target audio, which increases the interaction with the user to increase the user's product stickiness to the album in this application.

[0041] S40: Generate a target video according to the target story script.

[0042] Specifically, after generating a target story script according to the story background, dynamic characters and target audio, a target video is synthesized based on the story background, character dynamics and target audio of the story script.

[0043] In one embodiment, generating a target video according to a target story script specifically includes: S41: generating a preliminary video according to the target story script; S42: Add background audio to the preliminary video to generate a target video.

[0044] Specifically, a preliminary video is synthesized based on the story background, character dynamics, and target audio of the story script, wherein the preliminary video refers to a video without background music added, wherein the background audio is generated based on the target story script.

[0045] For example, an existing photo shows a person running on a riverside trail, and there are several trees by the river. In this photo, the image background is the trees by the river and the river. Based on the direction of the water flow in the river and the growth direction of the leaves in the picture, a story background is generated, and the static river in the picture is added with water flow dynamics. Based on the growth of the leaves, the dynamics of the wind blowing the leaves are added. For the person in the picture, it is extracted that he is running, and the person features of the person in the picture are extracted. Based on the direction of the person's running and the running posture in the photo, a dynamic effect is added to the photo, so that the person in the picture can look like he is continuing to move in the video. The action of the person in the image is supplemented or added to make the video look more complete. At the same time, the audio uploaded by the user is analyzed, the user's audio features are extracted, and the user's audio features are obtained based on the audio features. The data includes the frequency of the audio, the user's voice habits and other user audio timbre features, and based on the user's audio timbre, the video content explanation of the user's audio is added at an appropriate position in the video. After synthesizing the preliminary video according to the background, the dynamics of the characters, and the target audio, background music is added according to the overall style of the synthesized preliminary video to form the final target video, and the synthesized target video is automatically saved. The above examples are for illustration only and are not intended to be limiting.

[0046] In this embodiment, a preliminary video is generated according to the target story script, and background audio is added to the preliminary video to generate a target video. This application generates a target video and adds background music to the target video based on the background in the picture, the characteristics of the characters in the picture, and the user audio, so that the static photos in the original mobile device album are turned into dynamic videos, increasing the interaction between the mobile device album and the user, and increasing the product stickiness between the user and the mobile device album and the user.

[0047] In one embodiment, generating a target video according to a target story script includes: If you are not satisfied with the generated target video, you can edit and modify the generated target video again through a voice intelligent assistant or manual operation, and back up and store the modified target video.

[0048] Specifically, after the target video is generated according to the background in the picture, the character features in the picture, and the user audio, if the user feels that it does not meet his expectations after watching it, he can use the voice intelligent assistant or manual operation to edit and modify the generated target video again, and back up and store the modified target video, wherein the modified range includes the dynamic amplitude of the background in the video, the movement amplitude of the character feature action, the displacement size of the character feature movement, the size of the target audio, and whether the video dynamics in the generated target video move, whether the character features move, whether to add the target audio, etc. can all be modified. After the modification, the user can modify the generated target video at any time until the user is satisfied. The video that the user confirms to be satisfied is saved.

[0049] Exemplarily, if the user thinks that the background dynamics in the generated target video are too fast, the dynamic frequency of the video background can be adjusted separately. If the user thinks that the amplitude of the movement of the character features in the generated target video is too large, the amplitude of the character movement in the video can be adjusted separately. If the user thinks that the volume or speech rate of the target audio in the generated target video does not meet the requirements, they can be modified separately or in combination. For example, the background dynamics and the amplitude of the character movement can be modified at the same time, the volume or speech rate of the target audio and the amplitude of the character movement can be modified at the same time, the background dynamics and the volume or speech rate of the target audio can be modified at the same time, and the background dynamics, the amplitude of the character movement and the volume or speech rate of the target audio can be modified at the same time. The above examples are only for illustrative purposes. After the target video is generated, the unsatisfactory parts of the video can be modified, that is, the background dynamics and the amplitude of the character movement in the video, the volume or speech rate of the target audio can be modified in combination or separately, and there is no specific limitation on the amplitude of the modification and the method of modification.

[0050] In this embodiment, the above-mentioned video synthesis method, device, electronic device and storage medium collect images and audio uploaded by users, extract image background, character features and audio features, generate a target story script based on the extracted image background, character features and audio features, and generate a target video based on the target story script. By acquiring the user's images and audio, extracting the user's audio features and picture background and character features, generating a story script based on the user's audio features and picture background and character features, generating a target video that belongs only to the user, and automatically turning the originally static photos into dynamic videos through the processing of artificial intelligence technology, which increases the fun of the photos and enhances the interactivity between the album and the user.

[0051] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0052] In one embodiment, a video synthesis device is provided, which corresponds one-to-one to the video synthesis method in the above embodiment. Figure 2 As shown, the video synthesis device includes a collection module, an extraction module, a processing module and a video generation module. The detailed description of each functional module is as follows: The acquisition module is used to collect images and audio uploaded by users; An extraction module, used to extract image background, character features and audio features; A processing module, used for generating a target story script according to the extracted image background, character features and audio features; The video generation module is used to generate a target video according to a target story script.

[0053] Specifically, images and audio uploaded by the user are collected. The uploaded images can be pictures taken by the user through a mobile terminal, can be user pictures stored in the mobile terminal device, or can be pictures obtained by other methods that can obtain user photos. The specific method of obtaining the photos is not specifically limited. The uploaded user audio is obtained by collecting the user audio using an audio collection device. The audio collection device can be an audio collection device that comes with the mobile device, or an external audio collection device. The specific method of collecting audio is also not limited. After obtaining the images and audio uploaded by the user, it is necessary to process the images uploaded by the user, divide the images uploaded by the user into image background and character parts, and extract features from each part to ensure that the script is designed according to the images uploaded by the user.

[0054] After the user has collected audio, the audio features of the user are extracted according to the audio uploaded by the user. The audio features include the audio timbre, audio frequency and other audio features of the user when speaking. The user audio features are extracted to facilitate adding the user audio to the synthesized video. After extracting the background, character features and audio features of the image uploaded by the user, the target story script is generated according to the image background, the character features and the audio features. After the target story script is generated according to the story background, dynamic characters and target audio, the target video is synthesized according to the story background, character dynamics and target audio of the story script.

[0055] For the specific definition of the video synthesis device, please refer to the definition of the video synthesis method above, which will not be repeated here. Each module in the above-mentioned video synthesis device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the electronic device in the form of hardware, or can be stored in the memory of the electronic device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0056] In one embodiment, an electronic device is provided. The electronic device may be a server, and its internal structure diagram may be as follows: Figure 3 As shown. The electronic device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the electronic device is used to realize that the originally static photos are automatically converted into dynamic videos through the processing of artificial intelligence technology. The network interface of the electronic device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a video synthesis method is implemented.

[0057] In one embodiment, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the video synthesis method in the above embodiment is implemented, for example Figure 1 A flow chart of a video synthesis method in an embodiment of the present invention, or Figures 2 to 3 Alternatively, when the processor executes the computer program, the functions of each module / unit in the embodiment of the video synthesis device are realized, for example Figure 3 The functions of the video synthesis shown are not described here in detail to avoid repetition.

[0058] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the video synthesis method in the above embodiment is implemented, for example Figure 1 A flow chart of a video synthesis method in an embodiment of the present invention, or Figures 2 to 3 Alternatively, when the computer program is executed by a processor, the functions of each module / unit in the above-mentioned video synthesis device embodiment are realized, for example Figure 3 The functions of the video synthesis shown are not described here in detail to avoid repetition.

[0059] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0060] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0061] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A video synthesis method, characterized in that: include: Collect images and audio uploaded by users; Extract image background, character features and audio features from the image; Generate a target story script according to the extracted image background, character features, and audio features; Generate a target video based on the target story script.

2. The video synthesis method according to claim 1, characterized in that: Collect the audio uploaded by the user, then include: According to the audio uploaded by the user, the user's voice frequency and voice timbre are obtained, and the user's voice frequency and voice timbre are automatically saved.

3. The video synthesis method according to claim 1, characterized in that: Generate a target story script according to the extracted image background, character features, and audio features, including: Analyzing the image background using artificial intelligence technology to generate a story background based on the image background; Generate a dynamic character and target audio according to the extracted character features and audio features; A target story script is generated based on the acquired story background, the dynamic characters and the target audio.

4. The video synthesis method according to claim 1, characterized in that: Generate a target video based on the target story script, including: Generate a preliminary video according to the target story script; Background audio is added to the preliminary video to generate a target video.

5. The video synthesis method according to claim 3, characterized in that: The target audio is obtained by using audio synthesis technology.

6. The video synthesis method according to claim 4, characterized in that: The background audio is generated according to the target story script, including: Obtain the story background, dynamic characters and target audio generated by the target story script; Generate appropriate background audio based on the story background, dynamic characters and target audio generated by the story script.

7. The video synthesis method according to claim 1, characterized in that: Generate a target video based on the target story script, then include: If you are not satisfied with the generated target video, you can edit and modify the generated target video again through a voice intelligent assistant or manual operation, and back up and store the modified target video.

8. A video synthesis device, characterized in that: include: The acquisition module is used to collect images and audio uploaded by users; An extraction module, used to extract image background, character features and audio features; A processing module, used for generating a target story script according to the extracted image background, character features and audio features; The video generation module is used to generate a target video according to a target story script.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the video synthesis method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the video synthesis method according to any one of claims 1 to 7 is implemented.

Images