Video generation method and device

Abstract

The disclosed embodiment provides a video generation method and device, the method comprising: obtaining target audio; generating an image sequence according to feature information of the target audio and an image generation model, the image generation model being used to generate a corresponding image according to a randomly input vector; combining the target audio and the image sequence to generate a target video corresponding to the target audio. Thus, a video suitable for the audio is generated, with richer expression forms and no copyright risk, and the efficiency of converting audio to video is improved.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of computer technology, and in particular, to a video generation method and device. Background Art

[0002] With the development of network technology, videos that integrate rich elements such as images, text, and sound have gradually become the mainstream expression method on the Internet.

[0003] In addition to video, audio is also a form of expression on the Internet. For example, users can upload songs they have recorded on social platforms, or share songs they have recently listened to on social platforms to express their listening preferences, recent dynamics, etc. However, compared with video, audio has a single form of expression, and users often encounter the problem of not having video materials suitable for audio when uploading audio on video websites.

[0004] Therefore, how to convert audio into video with richer content is a problem that needs to be solved at present. Summary of the invention

[0005] The embodiments of the present disclosure provide a video generation method and device to realize the conversion of audio into a video with richer content.

[0006] In a first aspect, an embodiment of the present disclosure provides a video generation method, including:

[0007] Get the target audio;

[0008] Generate an image sequence according to the feature information of the target audio and an image generation model, wherein the image generation model is used to generate a corresponding image according to a randomly input vector;

[0009] The target audio and the image sequence are combined to generate a target video corresponding to the target audio.

[0010] In a second aspect, an embodiment of the present disclosure provides a model determination method, including:

[0011] Acquire training data, wherein the training data includes a plurality of images;

[0012] Based on the training data, training an image generation model multiple times, wherein the image generation model is used to generate a corresponding image according to a randomly input vector;

[0013] A training process of the image generation model includes:

[0014] randomly determining an input vector and randomly determining label data in the plurality of images;

[0015] Inputting the input vector into the image generation model to obtain an output image;

[0016] The image generation model is adjusted according to the difference between the output image and the label data.

[0017] In a third aspect, an embodiment of the present disclosure provides a video generating device, including:

[0018] An acquisition unit, used for acquiring target audio;

[0019] An image generation unit, configured to generate an image sequence according to the feature information of the target audio and an image generation model, wherein the image generation model is configured to generate a corresponding image according to a randomly input vector;

[0020] The video generation unit is used to combine the target audio and the image sequence to generate a target video corresponding to the target audio.

[0021] In a fourth aspect, an embodiment of the present disclosure provides a model determination device, including:

[0022] An acquisition unit, configured to acquire training data, wherein the training data includes a plurality of images;

[0023] A training unit, used for training an image generation model multiple times based on the training data, wherein the image generation model is used for generating a corresponding image according to a randomly input vector;

[0024] During a training process of the image generation model, the training unit is further used to:

[0025] randomly determining an input vector and randomly determining label data in the plurality of images;

[0026] Inputting the input vector into the image generation model to obtain an output image;

[0027] The image generation model is adjusted according to the difference between the output image and the label data.

[0028] In a fifth aspect, an embodiment of the present disclosure provides an electronic device, including: at least one processor and a memory;

[0029] The memory stores computer-executable instructions;

[0030] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the video generation method as described in the first aspect or various possible designs of the first aspect, or, so that the at least one processor executes the model determination method as described in the second aspect or various possible designs of the second aspect.

[0031] In a sixth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer execution instructions. When a processor executes the computer execution instructions, the video generation method as described in the first aspect or various possible designs of the first aspect is implemented, or the model determination method as described in the second aspect or various possible designs of the second aspect is implemented.

[0032] In the seventh aspect, according to one or more embodiments of the present disclosure, a computer program product is provided, wherein the computer program product includes computer execution instructions. When a processor executes the computer execution instructions, the video generation method as described in the first aspect or various possible designs of the first aspect is implemented, or the model determination method as described in the second aspect or various possible designs of the second aspect is implemented.

[0033] The video generation method and device provided in this embodiment generates an image sequence based on the characteristic information of the target audio and the image generation model, combines the target audio with the image sequence, and generates a target video corresponding to the audio, wherein the image generation model is used to generate a corresponding image according to a randomly input vector. Thus, by combining the characteristic information of the audio and the image generation model used to generate a corresponding image according to a randomly input vector, a video suitable for the audio, personalized, and free of copyright risks is generated for the audio, and the efficiency of converting audio to video is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0035] Figure 1 A schematic diagram of an application scenario to which the embodiments of the present disclosure are applicable;

[0036] Figure 2 A schematic diagram of the process of the video generation method provided in the embodiment of the present disclosure Figure 1 ;

[0037] Figure 3 A schematic diagram of the process of the video generation method provided in the embodiment of the present disclosure Figure 2 ;

[0038] Figure 4 A schematic diagram of the process of the video generation method provided in the embodiment of the present disclosure Figure 3 ;

[0039] Figure 5 Schematic diagram of the process of the model determination method provided in the embodiment of the present disclosure Figure 1 ;

[0040] Figure 6 A schematic diagram of a process of converting music into video provided by an embodiment of the present disclosure;

[0041] Figure 7 A structural block diagram of a video generation device provided in an embodiment of the present disclosure;

[0042] Figure 8 A structural block diagram of a model determination device provided in an embodiment of the present disclosure;

[0043] Fig. 9 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0045] In the prior art, audio is converted into video by users who manually collect images as video materials and arrange the image playback time based on the audio playback timeline. This method has the following shortcomings: 1. The entire process is time-consuming and inefficient; 2. The richness of video content is limited by the user's ability to collect and organize video materials, and the video quality is greatly affected by subjective factors; 3. It is difficult to produce a video that matches the style or type of audio; 4. Users manually collect video materials on the Internet, and need to pay attention to whether the video materials are authorized, which poses a copyright risk.

[0046] To solve the above problems, the embodiments of the present disclosure provide a video generation method and device, which generates an image sequence according to the characteristic information of the target audio and an image generation model for generating corresponding images according to a randomly input vector, and then combines the target audio with the image sequence to generate a target video corresponding to the target audio. It can be seen that the embodiments of the present disclosure have the following beneficial effects: 1. It effectively improves the efficiency of converting audio to video. The user only needs to input the target audio to obtain the target video corresponding to the target audio; 2. The quality of the video content is stable, and the requirements for the user's video production ability are low. It can provide audio-to-video services for a wide range of user groups and improve user experience; 3. In the process of generating the image sequence, the characteristic information of the target audio is referenced, and a video that matches the characteristic information of the audio can be generated, which improves the personalization and uniqueness of the video; 4. The image sequence as the video material is generated by the image generation model based on the randomly input vector, and there is no copyright risk.

[0047] refer to Figure 1 , Figure 1 The present invention is a schematic diagram of an application scenario applicable to the embodiments of the present disclosure.

[0048] like Figure 1 As shown, in this application scenario, the devices involved include a terminal 101 and a server 102. A user can input audio on the terminal 101, and the terminal 101 sends the audio input by the user to the server 102. The server 102 converts the audio into a video, sends the video to the terminal 101, and the terminal 101 displays the video to the user. The terminal 101 and the server 102 communicate, for example, through a network.

[0049] Taking the example of a user uploading audio on a video platform, the terminal 101 has an application for the video platform installed, and the server 102 is the background server (or product server) corresponding to the video platform. The user inputs the audio to be uploaded on the application on the terminal 101, and the terminal 101 sends the audio to the server 102. The server 102 converts the audio into video and returns the video to the terminal 101. The terminal 101 displays the video to the user. After the user confirms that the video meets his or her needs, he or she decides to upload the video.

[0050] When the computing capability of the terminal 101 is relatively strong, the terminal 101 may convert the audio into video.

[0051] The terminal may be a personal digital assistant (PDA), a handheld device (such as a smart phone, a tablet), a computing device (such as a personal computer (PC)), a vehicle-mounted device, a wearable device (such as a smart watch, a smart bracelet), and a smart home device (such as a smart display device). The server may be a distributed server, a centralized server, a cloud server, etc.

[0052] Hereinafter, multiple embodiments of the present disclosure are provided, wherein the execution subject of the multiple embodiments of the present disclosure may be an electronic device, and the electronic device may be a terminal or a server.

[0053] refer to Figure 2 , Figure 2 The video generation method provided in the embodiment of the present disclosure is shown in the following figure. Figure 1 .like Figure 2 As shown, the video generation method includes:

[0054] S201: Acquire target audio.

[0055] The number of target audios may be one or more, and the type of target audios may be songs, voices, etc. Here, there is no restriction on the type and format of the target audios.

[0056] In one example, target audio input by a user is obtained, for example, music recorded or uploaded by a user on an application such as a music player or a video platform is obtained.

[0057] In another example, one or more target audios are obtained from an audio database. For example, the target audios may be obtained randomly from the audio database, or the target audios may be obtained in a storage order, or the target audios may be obtained in a user-specified order, or the target audios may be obtained as specified by the user. The audio database may be a local database or a cloud database.

[0058] In another example, the target audio from another device is obtained. For example, when the current execution subject is a server, the target audio from the terminal can be obtained.

[0059] In this embodiment, the target audio can be obtained from the video production request in response to the target audio, or the target audio can be obtained by using any of the above examples. The target audio production request can be triggered by a preset user operation. For example, in the application corresponding to the video platform, after uploading the target audio, the user clicks the "Make Video" button to trigger the target audio production request.

[0060] S202: Generate an image sequence according to feature information of the target audio and an image generation model, where the image generation model is used to generate a corresponding image according to a randomly input vector.

[0061] The characteristic information of the target audio may reflect one or more characteristics of the target audio.

[0062] Among them, the image generation model can be a deep learning model whose input data is a vector and whose output data is an image, so as to improve the image generation quality through the deep learning model.

[0063] In this embodiment, feature information of the target audio is extracted from the target audio. An input vector is randomly generated, and based on the feature information of the target audio and the randomly generated input vector, a plurality of images are generated using an image generation model to obtain an image sequence. In the process of generating images, the feature information of the target audio is used so that the image generated by the image generation model conforms to one or more features of the audio, thereby improving the degree of fit between the image and one or more features of the audio. The randomly generated input vector is used so that the images in the image sequence have randomness, thereby generating a unique, personalized, and target audio-suitable image sequence for the target audio and avoiding copyright risks.

[0064] S203: Combine the target audio and image sequence to generate a target video corresponding to the target audio.

[0065] In this embodiment, the play time of each frame of the target video can be arranged based on the play time axis of the target audio, and the target audio and the image sequence can be combined on the play time axis of the target audio to obtain the target video with the target audio as the video audio and the image sequence as the video screen. Thus, a unique, personalized video suitable for the target audio is generated for the audio.

[0066] In the disclosed embodiment, an image sequence is generated based on the characteristic information of the target audio and an image generation model for generating corresponding images according to a randomly input vector, and a target video is generated by combining the target audio and the image sequence. Thus, the efficiency of converting audio to video is improved, the requirements for the user's video production ability are reduced, and the audio-to-video service is provided to more users. Moreover, the converted video has the characteristics of being unique, personalized, and highly consistent with the audio, and avoids copyright risks.

[0067] Considering that audios of different scene types are suitable for images of different scene types, and limited by the training data, an image generation model has a better model effect when applied to generate images of a specific scene type, and is more capable of generating images with better quality and more accurate scene types. In view of this, the present disclosure provides an embodiment to improve the degree of fit between the converted video and audio in terms of scene type, that is, to achieve the conversion of audio into a video of the same or similar scene type.

[0068] refer to Figure 3 , Figure 3 The video generation method provided in the embodiment of the present disclosure is shown in the following figure. Figure 2 .like Figure 3 As shown, the video generation method includes:

[0069] S301: Acquire target audio.

[0070] The implementation principle and technical effects of S301 may refer to the aforementioned embodiments and will not be described in detail.

[0071] S302: Determine the target scene type to which the target audio belongs.

[0072] Among them, the scene type to which the audio belongs can reflect the characteristics of the audio from the perspective of the scene, making the audio have a sense of picture. For example, the scene types include starry sky scenes, natural scenes, fireworks scenes, etc. It can be seen that in the process of converting audio to video, using the scene type to which the audio belongs is conducive to converting a video with a higher degree of fit between the video picture and the audio.

[0073] In one example, a target scene type input by a user is obtained.

[0074] For example, the user inputs the target scene type to which the target audio belongs while inputting the target audio; for another example, after obtaining the target audio, the user is reminded to input the target scene type to which the target audio belongs, and then the target scene type input by the user is obtained.

[0075] In another example, the target scene type to which the target audio belongs can be determined through natural language processing and audio information of the target audio.

[0076] The audio information of the target audio includes one or more of the following: the name of the target audio, the introduction of the target audio, and the content of the target audio.

[0077] Specifically, a preset scene dictionary can be used to perform natural language processing on the audio information of the target audio, such as word segmentation, matching, etc., to extract words related to the scene type, and based on these words, the target scene type to which the target audio belongs can be obtained. Thus, the accuracy of determining the target scene type to which the target audio belongs can be improved by using the natural language processing method.

[0078] Taking songs as an example, the audio information of a song includes the song's name, introduction and lyrics. By using natural language processing methods, we can extract scene-related words such as "starry sky" and "moon" from the song's name, introduction and lyrics, and these words can constitute the song's scene information.

[0079] In yet another example, among multiple scene types, the target scene type is determined to be a scene type that matches identification information of the target audio.

[0080] Among them, multiple scene types are pre-set, such as starry sky scene, natural scene, and fireworks scene. The identification information of the target audio includes the name, label words and / or logo image of the target audio. For example, for some songs related to starry sky scenes, on the music playback platform, users may add label words such as "starry sky", "moon", and "stars" to these songs; the logo image is the cover image of the target audio.

[0081] Specifically, multiple scene types are matched with the identification information of the target audio to obtain the matching degree between the multiple scene types and the identification information of the target audio, and the target scene type is determined to be the scene type with the highest matching degree with the identification information of the target audio among the multiple scene types. Among them, matching the identification information of the multiple scene types with the target audio includes at least one of the following: matching the names of the multiple scene types with the names and / or identification words of the target audio, and matching the scene images of the multiple scene types with the identification images of the target audio. Thus, the accuracy of determining the target scene type to which the target audio belongs is improved by means of image-text matching.

[0082] S303: Determine an image generation model corresponding to the target scene type, where the image generation model corresponding to the target scene type is used to generate an image corresponding to the target scene type according to the randomly input vector.

[0083] Among them, image scene models corresponding to multiple scene types can be pre-trained to obtain corresponding relationships between multiple scene types and image generation models. In this corresponding relationship, the image generation model corresponding to the scene type is used to generate images corresponding to the scene type.

[0084] For example, the image generation model corresponding to the starry sky scene is used to generate images corresponding to the starry sky scene; the image generation model corresponding to the natural scene is used to generate images corresponding to the natural scene; and the image generation model corresponding to the fireworks scene is used to generate images corresponding to the fireworks scene.

[0085] In this embodiment, the image generation model corresponding to the target scene type can be determined based on the correspondence between the scene type and the image generation model. Specifically, in the correspondence, the target scene type is searched, and then the image generation model corresponding to the target scene type is determined, thereby improving the fit between the image and the audio generated by the image generation model in the scene type, and then improving the fit between the subsequently generated video and the audio in the scene type, thereby improving the video generation effect.

[0086] Optionally, the model structures of the image generation models corresponding to different scene types are the same.

[0087] Optionally, the model structures of image generation models corresponding to different scene types are different, so that the corresponding image generation models can be designed to adapt to the actual conditions of different scenes and improve the model performance of the image generation models. For example, the image generation model in the face scene includes a network structure for face recognition, while the image generation model in other scenes does not need to design this network structure.

[0088] S304: Generate an image sequence according to the feature information of the target audio and the image generation model.

[0089] S305: Combine the target audio and image sequence to generate a target video corresponding to the target audio.

[0090] The implementation principles and technical effects of S304 to S305 may refer to the aforementioned embodiments and will not be described in detail.

[0091] In the disclosed embodiment, an image sequence is generated based on the characteristic information of the target audio and the image generation model corresponding to the target scene type to which the target audio belongs, and a target video is generated by combining the target audio and the image sequence. Thus, on the basis of the beneficial effects of the aforementioned embodiment, the degree of fit between the target video and the target audio is further improved, so that the video screen of the target video can more accurately reflect the scene to which the audio belongs.

[0092] Based on any of the foregoing embodiments, the present disclosure provides another embodiment.

[0093] refer to Figure 4 , Figure 4 The video generation method provided in the embodiment of the present disclosure is shown in the following figure. Figure 3 . Such as Figure 4 As shown, the video generation method includes:

[0094] S401: Acquire target audio.

[0095] The implementation principle and technical effects of S401 may refer to the aforementioned embodiments and will not be described in detail.

[0096] S402: Randomly determine an initial input vector.

[0097] In this embodiment, the initial input vector may be randomly determined according to a preset data distribution, for example, according to a normal distribution, wherein the input vector is, for example, a one-dimensional vector with a vector size of 1*512.

[0098] S403. Perform multiple image generation through an image generation model according to the input vector and feature information to obtain an image sequence. Feature information of the target audio in the multiple image generation is used to update the input vector.

[0099] In this embodiment, in the first image generation, the initial input vector is input into the image generation model to obtain the first image output by the image generation model. Then, in the second image generation, the input vector is updated according to the feature information of the target audio, and the updated input vector is the input vector in the second image generation. The input vector is input into the image generation model to obtain the second image output by the image model. In the third image generation, the input vector is updated again according to the feature information of the target audio. This cycle is repeated to obtain multiple images. Thus, by updating the input vector using the feature information of the target audio in each image generation process, the image generated based on the random input vector can reflect the feature information of the audio.

[0100] In one example, when updating the input vector according to the feature information of the target audio, the target change value can be determined based on the feature information of the target feature, and the input vector can be increased or decreased according to the target change value. For example, the target vector is a certain feature value of the target audio, and the feature value is added to the input vector, or the feature value is subtracted from the input vector.

[0101] In another example, considering that the changes of audio and video are coherent or continuous, in order to ensure the coherence or continuity of the changes of each frame image in the target video and improve the consistency of the changes of audio and video, the update direction of the input vector is consistent in the multiple image generation processes. Figure 4 As shown, a possible implementation method of S403 includes: determining a direction vector; performing multiple image generation through an image generation model according to the input vector, feature information of the audio and the direction vector to obtain an image sequence, wherein the direction vector is related to the update direction of the input vector in each image generation.

[0102] Optionally, when determining the direction vector, the direction vector is determined randomly, so that the change direction of the video picture in the same video is consistent, and the change direction of the video picture in different videos is random, further increasing the randomness of the video obtained by audio conversion.

[0103] Optionally, when determining the direction vector, obtain the direction vector input by the user.

[0104] Optionally, when determining the direction vector, a preset direction vector is obtained.

[0105] Specifically, the direction vector is determined first. When the input vector is updated, the change amount of the input vector is determined according to the characteristic information of the audio, and the input vector is updated by combining the change amount and the direction vector. Thus, the update direction of the input vector in multiple image generation is consistent, so that the input vector changes continuously in multiple image generation, and then the image output by the image generation model also changes continuously with the continuous change of the input vector, thereby improving the continuity of the change of the target video.

[0106] Optionally, the change in the input vector is proportional to the characteristic information of the audio, thereby improving the similarity in amplitude between the change in the input vector and the change in the characteristic information of the audio.

[0107] In one example, the characteristic information of the target audio changes over time. At this time, the above-mentioned one-time image generation process of multiple image generation includes: determining the characteristic information of the target audio in the target unit time, wherein the target unit time is the unit time corresponding to the number of times the current image is generated; and updating the input vector according to the characteristic information and direction vector of the target audio in the unit time. Thus, based on the characteristic information of the target audio that changes over time, the consistency of the change of the characteristic information of the audio and the change of the input vector is improved, thereby improving the consistency of the change of the audio and the change of the video.

[0108] The consistency of the change of the characteristic information of the audio and the change of the input vector includes one or more of the following aspects: the consistency of the change of the characteristic information of the audio and the change of the input vector in time, and the consistency of the change of the characteristic information of the audio and the change of the input vector in the change amplitude. For example, if a certain characteristic value of the audio decreases at time a, the input vector also decreases at time a, and / or, if the change amplitude of a certain characteristic value of the audio is large, the change amplitude of the input vector is also large.

[0109] Similarly, the consistency of the audio change and the video change includes one or more of the following aspects: the consistency of the audio change and the video change in time, and the consistency of the audio change and the video change in amplitude. For example, when the audio melody changes, the video picture also changes, and when the audio melody changes significantly, the video picture also changes significantly.

[0110] Specifically, the correspondence between multiple unit times on the playback time axis of the target audio and the number of image generation times can be determined first, wherein on the playback time axis, the first unit time corresponds to the first image generation, the second unit time corresponds to the second image generation, and so on. At this time, the generation process of the image sequence includes: in the first image generation, the randomly determined initial input vector is input into the image generation model to obtain the first image; in the second image generation, the input vector is updated according to the feature information and direction vector of the target audio in the second unit time, and the updated input vector is input into the image generation model to obtain the second image; in the third image generation, the input vector is updated according to the feature information and direction vector of the target audio in the third unit time, and the updated input vector is input into the image generation model to obtain the third image; and so on. The total number of image generation times is equal to the total number of unit times. When the number of image generation times is greater than the total number, the image generation process is terminated to obtain an image sequence consisting of the first image, the second image, and so on.

[0111] Optionally, the characteristic information of the target audio includes the frequency of the target audio in multiple unit times. In this case, an image generation process includes: an image generation process of the above multiple image generation, including: determining the frequency of the target audio in the target unit time; updating the input vector according to the frequency and direction vector of the target audio in the target unit time. Among them, the frequency of the target audio in multiple unit times reflects the melody of the target audio, so that the picture of the video changes continuously with the change of the melody of the target audio, thereby improving the fit between the video and audio obtained by audio conversion and improving the video quality.

[0112] Further, updating the input vector according to the frequency and direction vector of the target audio in a unit time includes: updating the input vector according to a first proportionality coefficient, the frequency and direction vector of the target audio in a target unit time. The first proportionality coefficient and the frequency of the target audio in a target unit time are used to determine the amount of change before and after the input vector is updated, so that the input vector changes with the frequency of the target audio by means of the proportionality coefficient.

[0113] Furthermore, according to the first proportionality coefficient, the frequency of the target audio in the target unit time and the direction vector, the formula for updating the input vector can be expressed as: updated input vector = first proportionality coefficient * frequency * direction vector + input vector used for the last image generation.

[0114] Optionally, the characteristic information of the target audio includes the amplitude of the target audio in multiple unit times, thereby improving the fit between the video and audio obtained by audio conversion from the perspective of amplitude, so that the picture changes of the video reflect the amplitude changes of the audio. For example, when the amplitude of the audio changes greatly, the picture changes of the video are greater, and when the amplitude of the audio changes less, the picture changes of the video are less. Taking music as an example, the amplitude of the music reflects the changes in the drum beats of the music. When the drum beats of the music suddenly change, the picture changes of the video are greater, and when the drum beats of the music are relatively gentle, the picture changes of the video are smaller.

[0115] Among them, in each image generation, the process of updating the input vector based on the amplitude and direction vector of the target audio in the target unit time can refer to the process of updating the input vector based on the frequency and direction vector of the target audio in the target unit time in the aforementioned embodiment, and will not be repeated here.

[0116] Optionally, the characteristic information of the target audio includes the frequency of the target audio in multiple unit times and the amplitude of the target audio in multiple unit times. At this time, after updating the input vector based on the frequency and direction vector of the target audio in the target unit time, if the amplitude difference between the amplitude of the target audio in the target unit time and the amplitude of the previous unit time is greater than the difference threshold, the disturbance amount is determined according to the amplitude difference and the direction vector; according to the disturbance amount, the updated input vector is updated again. Thus, in the case where the amplitude of the target audio changes slightly, the input vector is updated based on the frequency of the target audio, and in the case where the amplitude of the target audio changes significantly (including a substantial decrease in the amplitude or a substantial increase in the amplitude), the input vector is updated based on the frequency and amplitude of the target audio, further improving the fit between the video and audio obtained by audio conversion, so that the video picture can reflect the frequency change and amplitude change of the audio, thereby improving the video quality.

[0117] The difference threshold is a preset threshold.

[0118] Among them, the amplitude difference determines the size of the disturbance, and the direction vector determines the direction of the disturbance.

[0119] Further, determining the disturbance amount according to the amplitude difference and the direction vector includes: updating the input vector again according to the second proportional coefficient, the amplitude difference and the direction vector. The second proportional coefficient and the amplitude difference are used to determine the magnitude of the disturbance amount, so that the input vector changes with the amplitude of the target audio by combining the proportional coefficient with the amplitude.

[0120] Furthermore, the total update formula of the input vector is expressed as L: updated input vector = amplitude difference * direction vector + proportional coefficient * frequency * direction vector + input vector used for the previous image generation.

[0121] As an example, taking music as an example, based on the frequency of the music input by the user, the input vector is continuously updated, and the corresponding picture sequence is continuously generated through the image generation model to form a video. Among them, for the position of the pulse-like drum beat mutation (large amplitude change) in the music, a disturbance proportional to the amplitude change (that is, the above-mentioned disturbance amount) can be given to the input vector, and the disturbance can be removed after the drum beat (the amplitude change is also large), and the control vector can be restored to the same state as before the pulse arrived.

[0122] S404: Combine the target audio and image sequence to generate a target video corresponding to the target audio.

[0123] The implementation principle and technical effects of S404 may refer to the aforementioned embodiments and will not be described in detail.

[0124] In the disclosed embodiments, an initial input vector is randomly determined, and an image sequence is generated through an image generation model based on the input vector and feature information of the target audio. The feature information of the target audio is used to update the input vector in each image generation. Thus, on the basis of the beneficial effects of any of the aforementioned embodiments, the picture of the video obtained by audio conversion changes with the feature changes of the audio, so that the picture of the video obtained by audio conversion can more vividly reflect the features of the audio.

[0125] Based on any of the foregoing embodiments, optionally, the image generation model is a generative adversarial model, and an image sequence is generated according to the feature information of the target audio and the image generation model, including: generating an image sequence according to the feature information of the target audio and a generator in the image generation model. The generative adversarial model can be used to generate realistic images, so using the generative adversarial model as an image generation model is conducive to improving the quality of image generation, thereby improving the quality of the video obtained by audio conversion.

[0126] Specifically, during the training process, the image generation model can be trained to improve the quality of images generated by the generator in the image generation model. During the application process, the image sequence can be generated by the generator in the trained image generation model based on the feature information of the target audio. The generation process of the image sequence can refer to the above-mentioned embodiment. No more details are given.

[0127] The following describes the training of the image generation model in any of the aforementioned embodiments through examples.

[0128] It should be noted that the application process of the image generation model (i.e., the aforementioned multiple embodiments) and the training process of the image generation model can be performed on the same electronic device or on different electronic devices. For example: the application process of the image generation model is performed on the terminal, and the training process of the image generation model is performed on the server; the application process and the training process of the image generation model are performed on the same or different servers, or on the same or different terminals, which are not listed one by one here.

[0129] Reference Figure 5 , Figure 5 Schematic diagram of the process of the model determination method provided in the embodiment of the present disclosure Figure 1 .

[0130] like Figure 5 As shown, the model determination method includes:

[0131] S501: Acquire training data, where the training data includes multiple images.

[0132] In this embodiment, a plurality of images used as training data in the image generation model training process may be obtained from a training database, wherein the training database includes one or more pre-collected images.

[0133] In one example, image generation models corresponding to different scene types may be trained, and the image generation models corresponding to the scene types are used to generate images under the scene types (refer to the aforementioned embodiments, for example Figure 2 In this case, S501 includes: obtaining multiple scene images under multiple scene types, and for each scene type, determining that the training data corresponding to the scene type includes multiple scene images under the scene type. The multiple scene images under the multiple scene types can be obtained from a database.

[0134] For example, refer to Figure 6 , Figure 6 The following is a schematic diagram of a process for converting music into video according to an embodiment of the present disclosure. Figure 6 As shown, multiple images of starry sky scenes, multiple images of natural scenes, and multiple images of fireworks scenes are obtained, and the multiple images of starry sky scenes, multiple images of natural scenes, and multiple images of fireworks scenes are used to train image generation models corresponding to starry sky scenes, image generation models corresponding to natural scenes, and image generation models corresponding to fireworks scenes, respectively.

[0135] S502: Based on the training data, train the image generation model multiple times, where the image generation model is used to generate a corresponding image according to the randomly input vector.

[0136] Among them, Figure 5 As shown in Figure 1, the training process of the image generation model includes:

[0137] S5021, randomly determining an input vector, and randomly determining label data in a plurality of images;

[0138] S5022, inputting the input vector into the image generation model to obtain an output image;

[0139] S5023. Adjust the image generation model according to the difference between the output image and the label data.

[0140] In this embodiment, in each training process, an input vector is randomly determined, an image is selected from multiple images in the training data as label data, the input vector is input into the image generation model, an output image of the image generation model is obtained, and a loss value is determined according to the loss function of the image generation model, the output image and the label data, and the parameters of the image generation model are adjusted based on the loss value. In this way, multiple training processes are performed until the number of training times is greater than the number threshold, or until the loss value of the loss function is less than or equal to the loss threshold. Here, there is no restriction on the loss function.

[0141] In one example, when training image generation models corresponding to different scene types, the image generation model corresponding to each scene type can be trained based on the training data corresponding to the scene type, so that one image generation model is used to generate an image corresponding to one scene type. In each training process, for each scene type, a scene image under the scene type is selected from the training data corresponding to the scene type as label data.

[0142] In one example, the input vector may be randomly determined based on a preset data distribution, for example, based on a normal distribution.

[0143] In one example, a sampling method without replacement can be used to select label data from multiple images, thereby avoiding the situation where different input vectors extract the same image as label data at the same time, which affects the accuracy of the image generation model in generating different images according to different input vectors.

[0144] In one example, the image generation model is a generative adversarial model. In this case, during each training process, a randomly determined input vector is input into the generator in the image generation model to obtain the output image of the generator. The discriminator in the image generation model is used to discriminate the output image of the generator and the label data. In this way, the loss value corresponding to the generator and the loss value corresponding to the discriminator are obtained. According to these loss values, the generator and the discriminator are adjusted respectively. In this way, during the training process, the output image of the image generation model is continuously close to the label data, and finally, a trained image generation model is obtained.

[0145] It should be noted that the image generation model trained in this embodiment can be applied to the video generation method provided in any of the aforementioned embodiments to realize the conversion of audio into video.

[0146] For example, Figure 6 As shown, during the application process, videos can be generated based on the trained image generation models corresponding to various scene types and the frequency and amplitude of music.

[0147] Corresponding to the video generation method of the above embodiment, Figure 7 The structure block diagram of the video generation device provided by the embodiment of the present disclosure. For the convenience of explanation, only the part related to the embodiment of the present disclosure is shown. Figure 7 The video generating device includes: an acquiring unit 701, an image generating unit 702 and a video generating unit 703.

[0148] The acquisition unit 701 is used to acquire target audio.

[0149] The image generation unit 702 is used to generate an image sequence according to the feature information of the target audio and the image generation model, where the image generation model is used to generate a corresponding image according to a randomly input vector.

[0150] The video generation unit 703 is used to combine the target audio and the image sequence to generate a target video corresponding to the target audio.

[0151] In some embodiments, the video generation device also includes a determination unit 704, which is used to: determine the target scene type to which the target audio belongs; determine an image generation model corresponding to the target scene type, and the image generation model corresponding to the target scene type is used to generate an image of the corresponding target scene type according to a randomly input vector.

[0152] In some embodiments, the determination unit 704 is further configured to: determine, among multiple scene types, the target scene type as a scene type that matches the identification information of the target audio.

[0153] In some embodiments, the determination unit 704 is further configured to: determine the image generation model corresponding to the target scene type based on the correspondence between the scene type and the image generation model.

[0154] In some embodiments, the image generation unit 702 is further used to: randomly determine an initial input vector; perform multiple image generation through an image generation model based on the input vector and feature information to obtain an image sequence, wherein, in the multiple image generation, the feature information of the target audio is used to update the input vector.

[0155] In some embodiments, the update direction of the input vector in multiple image generations is consistent, and the image generation unit 702 is also used to: determine the direction vector; perform multiple image generation through the image generation model according to the input vector, feature information and the direction vector to obtain an image sequence, wherein the direction vector is related to the update direction of the input vector in each image generation.

[0156] In some embodiments, the feature information includes the frequency of the target audio in multiple unit times. The image generation unit 702 is used, during one image generation process among multiple image generation processes, to: determine the frequency of the target audio in a target unit time, wherein the target unit time is a unit time corresponding to the current number of image generation times; update the input vector according to the frequency and direction vector of the target audio in the target unit time; and input the updated input vector into the image generation model to obtain the current frame image.

[0157] In some embodiments, the feature information also includes the amplitude of the target audio in multiple unit times, and the image generation unit 702 is also used to: if the amplitude difference between the target audio in the target unit time and the amplitude in the previous unit time is greater than a difference threshold, determine the disturbance amount based on the amplitude difference and the direction vector; and update the updated input vector again based on the disturbance amount.

[0158] In some embodiments, the image generation model is a generative adversarial model, and the image generation unit 702 is further used to generate an image sequence according to feature information of the target audio and a generator in the image generation model.

[0159] The video generating device provided in this embodiment can be used to execute the technical solutions of the above-mentioned embodiments related to the video generating method. Its implementation principles and technical effects are similar and will not be described in detail in this embodiment.

[0160] Corresponding to the video generation method of the above embodiment, Figure 8 A structural block diagram of a model determination device provided in an embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Figure 8 , the model determination device includes: an acquisition unit 801 and a training unit 802.

[0161] An acquisition unit 801 is used to acquire training data, where the training data includes a plurality of images;

[0162] A training unit 802, used for training the image generation model multiple times based on the training data, where the image generation model is used to generate a corresponding image according to a randomly input vector;

[0163] In a training process of the image generation model, the training unit 802 is also used to:

[0164] Randomly determine the input vector and randomly determine the label data in multiple images;

[0165] Input the input vector into the image generation model to obtain the output image;

[0166] Adjust the image generation model based on the difference between the output image and the label data.

[0167] In some embodiments, the acquisition unit 801 is further used to: acquire multiple scene images under multiple scene types; and for each scene type, determine that the training data corresponding to the scene type includes the multiple scene images under the scene type.

[0168] In some embodiments, the training unit 802 is further used to: for each scene type, based on the training data corresponding to the scene type, train an image generation model corresponding to the scene type.

[0169] In some embodiments, the training unit 802 is further used to: randomly determine an input vector based on a preset data distribution; and select label data from multiple images using a sampling method without replacement.

[0170] In some embodiments, the image generation model is a generative adversarial network model.

[0171] The model determination device provided in this embodiment can be used to execute the technical solutions of the above-mentioned embodiments related to the model determination method. Its implementation principles and technical effects are similar and will not be repeated in this embodiment.

[0172] refer to Fig. 9 , which shows a schematic diagram of the structure of an electronic device 900 suitable for implementing the embodiment of the present disclosure, and the electronic device 900 may be a terminal device or a server. The terminal device may include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (Portable Android Devices, PADs), portable multimedia players (PMPs), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig. 9 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0173] like Fig. 9As shown, the electronic device 900 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage device 908 to a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the electronic device 900 are also stored. The processing device 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0174] Typically, the following devices may be connected to the I / O interface 905: input devices 906 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 907 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 908 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 909. The communication device 909 may allow the electronic device 900 to communicate with other devices wirelessly or by wire to exchange data. Although Fig. 9 The electronic device 900 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0175] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 909, or installed from a storage device 908, or installed from a ROM 902. When the computer program is executed by the processing device 901, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0176] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0177] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0178] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0179] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0180] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0181] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a unit does not limit the unit itself in some cases. For example, an acquisition unit may also be described as a "unit for acquiring target audio."

[0182] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0183] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0184] In a first aspect, according to one or more embodiments of the present disclosure, a video generation method is provided, comprising: obtaining target audio; generating an image sequence based on feature information of the target audio and an image generation model, wherein the image generation model is used to generate corresponding images based on a randomly input vector; and combining the target audio and the image sequence to generate a target video corresponding to the target audio.

[0185] According to one or more embodiments of the present disclosure, before generating an image sequence based on the feature information of the target audio and the image generation model, the method further includes: determining a target scene type to which the target audio belongs; determining an image generation model corresponding to the target scene type, wherein the image generation model corresponding to the target scene type is used to generate an image corresponding to the target scene type based on a randomly input vector.

[0186] According to one or more embodiments of the present disclosure, determining the target scene type to which the target audio belongs includes: determining, among multiple scene types, that the target scene type is a scene type that matches identification information of the target audio.

[0187] According to one or more embodiments of the present disclosure, the determining the image generation model corresponding to the target scene type includes: determining the image generation model corresponding to the target scene type based on a correspondence between scene types and image generation models.

[0188] According to one or more embodiments of the present disclosure, generating an image sequence based on the feature information of the target audio and an image generation model includes: randomly determining an initial input vector; performing multiple image generation through the image generation model based on the input vector and the feature information to obtain the image sequence, wherein, in the multiple image generation, the feature information of the target audio is used to update the input vector.

[0189] According to one or more embodiments of the present disclosure, the updating direction of the input vector in the multiple image generation is consistent, and the multiple image generation is performed by the image generation model based on the input vector and the feature information to obtain the image sequence, including: determining a direction vector; performing multiple image generation by the image generation model based on the input vector, the feature information and the direction vector to obtain the image sequence, wherein the direction vector is related to the updating direction of the input vector in each image generation.

[0190] According to one or more embodiments of the present disclosure, the feature information includes the frequency of the target audio in multiple unit times, and an image generation process in the multiple image generation includes: determining the frequency of the target audio in a target unit time, wherein the target unit time is a unit time corresponding to the current number of image generation times; updating the input vector according to the frequency of the target audio in the target unit time and the direction vector; and inputting the updated input vector into the image generation model to obtain the current frame image.

[0191] According to one or more embodiments of the present disclosure, the feature information also includes the amplitude of the target audio in multiple unit times, and after updating the input vector according to the frequency of the target audio in the target unit time and the direction vector, it also includes: if the amplitude difference between the amplitude of the target audio in the target unit time and the amplitude of the previous unit time is greater than a difference threshold, determining a disturbance amount according to the amplitude difference and the direction vector; and updating the updated input vector again according to the disturbance amount.

[0192] According to one or more embodiments of the present disclosure, the image generation model is a generative adversarial model, and the generating of the image sequence according to the feature information of the target audio and the image generation model includes: generating the image sequence according to the feature information of the target audio and a generator in the image generation model.

[0193] In a second aspect, according to one or more embodiments of the present disclosure, a model determination method is provided, comprising: obtaining training data, the training data comprising a plurality of images; based on the training data, training an image generation model multiple times, the image generation model being used to generate corresponding images according to a randomly input vector. A training process of the image generation model comprises: randomly determining an input vector and randomly determining label data in the plurality of images; inputting the input vector into the image generation model to obtain an output image; and adjusting the image generation model according to the difference between the output image and the label data.

[0194] According to one or more embodiments of the present disclosure, the acquiring of training data includes: acquiring multiple scene images under multiple scene types; and for each scene type, determining that the training data corresponding to the scene type includes the multiple scene images under the scene type.

[0195] According to one or more embodiments of the present disclosure, the training of the image generation model based on the training data includes: for each scene type, based on the training data corresponding to the scene type, training the image generation model corresponding to the scene type.

[0196] According to one or more embodiments of the present disclosure, the randomly determining the input vector and the randomly determining the label data in the multiple images include: randomly determining the input vector based on a preset data distribution; and selecting the label data in the multiple images using a sampling method without replacement.

[0197] According to one or more embodiments of the present disclosure, the image generation model is a generative adversarial network model.

[0198] In a third aspect, according to one or more embodiments of the present disclosure, a video generation device is provided, comprising: an acquisition unit for acquiring target audio; an image generation unit for generating an image sequence based on feature information of the target audio and an image generation model, wherein the image generation model is used to generate a corresponding image based on a randomly input vector; and a video generation unit for combining the target audio and the image sequence to generate a target video corresponding to the target audio.

[0199] According to one or more embodiments of the present disclosure, the video generation device also includes a determination unit, which is used to: determine the target scene type to which the target audio belongs; determine an image generation model corresponding to the target scene type, and the image generation model corresponding to the target scene type is used to generate an image corresponding to the target scene type based on a randomly input vector.

[0200] According to one or more embodiments of the present disclosure, the determining unit is further configured to: determine, among multiple scene types, that the target scene type is a scene type that matches identification information of the target audio.

[0201] According to one or more embodiments of the present disclosure, the determining unit is further configured to: determine the image generation model corresponding to the target scene type based on a correspondence between the scene type and the image generation model.

[0202] According to one or more embodiments of the present disclosure, the image generation unit is also used to: randomly determine an initial input vector; perform multiple image generation through the image generation model based on the input vector and the feature information to obtain the image sequence, wherein, in the multiple image generation, the feature information of the target audio is used to update the input vector.

[0203] According to one or more embodiments of the present disclosure, the update direction of the input vector in the multiple image generations is consistent, and the image generation unit is further used to: determine a direction vector; perform multiple image generation through the image generation model according to the input vector, the feature information and the direction vector to obtain the image sequence, wherein the direction vector is related to the update direction of the input vector in each image generation.

[0204] According to one or more embodiments of the present disclosure, the feature information includes the frequency of the target audio in multiple unit times, and the image generation unit is used, during one image generation process among the multiple image generation processes, to: determine the frequency of the target audio in a target unit time, wherein the target unit time is a unit time corresponding to the current number of image generation times; update the input vector according to the frequency of the target audio in the target unit time and the direction vector; and input the updated input vector into the image generation model to obtain the current frame image.

[0205] According to one or more embodiments of the present disclosure, the feature information also includes the amplitude of the target audio in multiple unit times, and the image generation unit is further used to: if the amplitude difference between the amplitude of the target audio in the target unit time and the amplitude of the previous unit time is greater than a difference threshold, determine the disturbance amount based on the amplitude difference and the direction vector; and update the updated input vector again based on the disturbance amount.

[0206] According to one or more embodiments of the present disclosure, the image generation model is a generative adversarial model, and the image generation unit is further used to generate the image sequence according to feature information of the target audio and a generator in the image generation model.

[0207] In a fourth aspect, according to one or more embodiments of the present disclosure, a model determination device is provided, comprising: an acquisition unit for acquiring training data, wherein the training data comprises a plurality of images; a training unit for training an image generation model multiple times based on the training data, wherein the image generation model is used to generate corresponding images according to a randomly input vector. During one training process of the image generation model, the training unit is further used to: randomly determine an input vector and randomly determine label data in the plurality of images; input the input vector into the image generation model to obtain an output image; and adjust the image generation model according to the difference between the output image and the label data.

[0208] According to one or more embodiments of the present disclosure, the acquisition unit is further used to: acquire multiple scene images under multiple scene types; and for each scene type, determine that the training data corresponding to the scene type includes the multiple scene images under the scene type.

[0209] According to one or more embodiments of the present disclosure, the training unit is further used to: for each scene type, train an image generation model corresponding to the scene type based on the training data corresponding to the scene type.

[0210] According to one or more embodiments of the present disclosure, the training unit is further used to: randomly determine an input vector based on a preset data distribution; and select the label data from the multiple images using a sampling method without replacement.

[0211] According to one or more embodiments of the present disclosure, the image generation model is a generative adversarial network model.

[0212] In a fifth aspect, according to one or more embodiments of the present disclosure, there is provided an electronic device, comprising: at least one processor and a memory;

[0213] The memory stores computer-executable instructions;

[0214] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the video generation method described in the first aspect or various possible designs of the first aspect, or, so that the at least one processor executes the model determination method described in the second aspect or various possible designs of the second aspect.

[0215] In a sixth aspect, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer execution instructions. When a processor executes the computer execution instructions, the video generation method described in the first aspect or various possible designs of the first aspect is implemented, or the model determination method described in the second aspect or various possible designs of the second aspect is implemented.

[0216] In the seventh aspect, according to one or more embodiments of the present disclosure, a computer program product is provided, wherein the computer program product includes computer execution instructions. When a processor executes the computer execution instructions, the video generation method as described in the first aspect or various possible designs of the first aspect is implemented, or the model determination method as described in the second aspect or various possible designs of the second aspect is implemented.

[0217] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0218] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0219] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A video generation method, comprising: Get the target audio; Generate an image sequence according to the feature information of the target audio and an image generation model, wherein the image generation model is used to generate a corresponding image according to a randomly input vector; Combining the target audio and the image sequence to generate a target video corresponding to the target audio; Among them, generating an image sequence according to the feature information of the target audio and the image generation model includes: randomly determining an initial input vector; performing multiple image generation through the image generation model according to the input vector and the feature information to obtain the image sequence, wherein, in the multiple image generation, the feature information of the target audio is used to update the input vector; the feature information includes the frequency of the target audio in multiple unit times, and an image generation process in the multiple image generation includes: determining the frequency of the target audio in a target unit time, wherein the target unit time is a unit time corresponding to the current number of image generation times; updating the input vector according to the frequency and direction vector of the target audio in the target unit time; and inputting the updated input vector into the image generation model to obtain the current frame image.

2. The video generation method according to claim 1, before generating the image sequence based on the feature information of the target audio and the image generation model, further comprising: Determine the target scene type to which the target audio belongs; An image generation model corresponding to the target scene type is determined, where the image generation model corresponding to the target scene type is used to generate an image corresponding to the target scene type according to a randomly input vector.

3. The video generation method according to claim 2, wherein determining the target scene type to which the target audio belongs comprises: Among multiple scene types, the target scene type is determined to be a scene type that matches the identification information of the target audio.

4. The video generation method according to claim 2, wherein determining the image generation model corresponding to the target scene type comprises: Based on the correspondence between the scene type and the image generation model, the image generation model corresponding to the target scene type is determined.

5. The video generation method according to claim 1, wherein the updating direction of the input vector in the multiple image generation is consistent, and the image generation model is used to generate multiple images based on the input vector and the feature information to obtain the image sequence, comprising: Determine the direction vector; The image sequence is obtained by performing multiple image generation through the image generation model according to the input vector, the feature information and the direction vector, wherein the direction vector is related to the update direction of the input vector in each image generation.

6. The video generation method according to claim 1, wherein the feature information further comprises the amplitude of the target audio in a plurality of unit times, and after updating the input vector according to the frequency of the target audio in a target unit time and the direction vector, the method further comprises: If the amplitude difference between the amplitude of the target audio in the target unit time and the amplitude of the previous unit time is greater than a difference threshold, determining a disturbance amount according to the amplitude difference and the direction vector; The updated input vector is updated again according to the disturbance amount.

7. The video generation method according to any one of claims 1 to 4, wherein the image generation model is a generative adversarial model, and the step of generating an image sequence according to the feature information of the target audio and the image generation model comprises: The image sequence is generated according to the feature information of the target audio and the generator in the image generation model.

8. A model determination method, applied to the image generation model according to any one of claims 1 to 7, comprising: Acquire training data, wherein the training data includes a plurality of images; Based on the training data, training an image generation model multiple times, wherein the image generation model is used to generate a corresponding image according to a randomly input vector; A training process of the image generation model includes: randomly determining an input vector and randomly determining label data in the plurality of images; Inputting the input vector into the image generation model to obtain an output image; The image generation model is adjusted according to the difference between the output image and the label data.

9. The model determination method according to claim 8, wherein obtaining training data comprises: Acquire multiple scene images under multiple scene types; For each scene type, it is determined that the training data corresponding to the scene type includes a plurality of scene images under the scene type.

10. The model determination method according to claim 9, wherein the step of training the image generation model based on the training data comprises: For each scene type, an image generation model corresponding to the scene type is trained based on the training data corresponding to the scene type.

11. The model determination method according to any one of claims 8 to 10, wherein the randomly determining the input vector and randomly determining the label data in the plurality of images comprises: Based on the preset data distribution, the input vector is randomly determined; The label data is selected from the multiple images by adopting a sampling method without replacement.

12. According to the model determination method according to any one of claims 8 to 10, the image generation model is a generative adversarial network model.

13. A video generating device, comprising: An acquisition unit, used for acquiring target audio; An image generation unit, configured to generate an image sequence according to the feature information of the target audio and an image generation model, wherein the image generation model is configured to generate a corresponding image according to a randomly input vector; A video generation unit, used to combine the target audio and the image sequence to generate a target video corresponding to the target audio; Among them, generating an image sequence according to the feature information of the target audio and the image generation model includes: randomly determining an initial input vector; performing multiple image generation through the image generation model according to the input vector and the feature information to obtain the image sequence, wherein, in the multiple image generation, the feature information of the target audio is used to update the input vector; the feature information includes the frequency of the target audio in multiple unit times, and an image generation process in the multiple image generation includes: determining the frequency of the target audio in a target unit time, wherein the target unit time is a unit time corresponding to the current number of image generation times; updating the input vector according to the frequency and direction vector of the target audio in the target unit time; and inputting the updated input vector into the image generation model to obtain the current frame image.

14. A model determination device, applied to the image generation model according to any one of claims 1 to 7, comprising: An acquisition unit, configured to acquire training data, wherein the training data includes a plurality of images; A training unit, used for training an image generation model multiple times based on the training data, wherein the image generation model is used for generating a corresponding image according to a randomly input vector; During a training process of the image generation model, the training unit is further used to: randomly determining an input vector and randomly determining label data in the plurality of images; Inputting the input vector into the image generation model to obtain an output image; The image generation model is adjusted according to the difference between the output image and the label data.

15. An electronic device, comprising: at least one processor and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the video generation method according to any one of claims 1 to 7, or executes the model determination method according to any one of claims 8 to 12.

16. A computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when a processor executes the computer-executable instructions, the video generation method as described in any one of claims 1 to 7 is implemented, or the model determination method as described in any one of claims 8 to 12 is executed.

17. A computer program product, comprising computer executable instructions, and when a processor executes the computer executable instructions, the video generation method according to any one of claims 1 to 7 is implemented, or the model determination method according to any one of claims 8 to 12 is executed.

Images