Video generation method and device, electronic equipment, medium and program product

Abstract

The invention discloses a video generation method and device, electronic equipment, a medium and a program product, and the method comprises the steps: obtaining an initial audio; segmenting the initial audio to obtain a plurality of audio segments; traversing the plurality of audio segments, and for any first audio segment in the plurality of audio segments, acquiring audio features of the first audio segment; selecting a target image matched with the first audio segment from a pre-acquired image library according to the audio feature of the first audio segment; and synthesizing the target image matched with each audio segment and the initial audio to obtain a target video, through the above process, the relevance between the audio segments and the images is improved, the adaptation degree of the audio and the images in the generated target video is higher, and the quality of the target video is improved.

Description

Technical Field

[0001] The present application belongs to the field of computer technology, and in particular, relates to a video generation method, device, electronic device, medium and program product. Background Art

[0002] With the continuous development of multimedia technology, the demand for the creation and generation of video content is growing. Traditional video production usually requires a lot of manpower, material resources and time investment, and the production process is complicated. People have an increasing demand for innovative and efficient video generation methods.

[0003] In related technologies, using audio information to drive digital images to generate videos has become a new research direction. However, in existing video generation technologies, the correlation between audio and image is weak, and the meanings expressed by the two are inconsistent. For example, the audio expresses a light and melodious style, but the image presents a sad scene. In other words, the compatibility between audio and image in the generated video is low, resulting in poor quality of the generated video. Summary of the invention

[0004] The embodiments of the present application provide a video generation method, device, electronic device, medium and program product, which can improve the adaptability of audio and image in the generated target video and improve the quality of the target video.

[0005] In a first aspect, an embodiment of the present application provides a video generation method, the method comprising:

[0006] Get the initial audio;

[0007] Segmenting the initial audio to obtain multiple audio segments;

[0008] Traversing the plurality of audio segments, and for any first audio segment among the plurality of audio segments, acquiring an audio feature of the first audio segment;

[0009] selecting, according to the audio features of the first audio segment, a target image matching the first audio segment from a pre-acquired image library;

[0010] A target image matched based on each of the audio segments is synthesized with the initial audio to obtain a target video.

[0011] In a second aspect, an embodiment of the present application provides a video generating device, the video generating device comprising:

[0012] A first acquisition module, used to acquire initial audio;

[0013] A segmentation module, used to segment the initial audio to obtain multiple audio segments;

[0014] A second acquisition module is used to traverse the multiple audio segments, and for any first audio segment among the multiple audio segments, acquire an audio feature of the first audio segment;

[0015] a matching module, configured to select a target image matching the first audio segment from a pre-acquired image library according to the audio features of the first audio segment;

[0016] A synthesis module is used to synthesize the target image matched with each audio segment with the initial audio to obtain a target video.

[0017] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory storing computer program instructions;

[0018] When the processor executes the computer program instructions, the video generating method as described in the first aspect is implemented.

[0019] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the video generation method described in the first aspect is implemented.

[0020] In a fifth aspect, an embodiment of the present application provides a computer program product. When instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the video generation method as described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0022] Figure 1 It is a flow chart of a video generation method provided in an embodiment of the present application;

[0023] Figure 2 is another flowchart of the video generation method provided in an embodiment of the present application;

[0024] Figure 3 is a schematic diagram of the structure of a video generating device provided in an embodiment of the present application;

[0025] Figure 4 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0027] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0028] In order to solve the problems in the prior art, the embodiments of the present application provide a video generation method, device, electronic device, medium and program product. The video generation method provided by the embodiments of the present application is first introduced below.

[0029] Figure 1 FIG. 1 is a flow chart of a video generation method provided by an embodiment of the present application. Figure 1 As shown, the video generation method provided in the embodiment of the present application is applied to an electronic device, and the video generation method includes the following steps 101 to 105, wherein:

[0030] Step 101, obtaining initial audio.

[0031] The initial audio may be audio captured from a song or film or television work, or may be audio downloaded from the Internet, or may be audio collected by a recording module of an electronic device, without limitation. The audio format of the initial audio may be a common audio format, such as MP3, WAV, FLAC, etc., without limitation.

[0032] In order to improve the audio effect and reduce noise interference, the acquired audio can be denoised, for example, using a wavelet transform method to remove background noise in the audio and improve the audio quality. It should be noted that the initial audio in this step can be the audio obtained after denoising.

[0033] Step 102: segment the initial audio to obtain multiple audio segments.

[0034] In order to facilitate more accurate feature extraction and image matching in the future, in this step, the initial audio is segmented, for example, the initial audio is segmented into multiple audio segments according to the pause points of the initial audio. The pause point refers to the point where the amplitude of the initial audio is significantly reduced in a short period of time, and the pause point can be detected by calculating the energy of the initial audio. If the energy of the audio segment segmented by a certain sampling point in the initial audio is lower than a preset threshold, it is considered that the location of the sampling point is in a pause state, and the sampling point is the pause point.

[0035] Step 103: traverse the multiple audio segments, and for any first audio segment among the multiple audio segments, obtain an audio feature of the first audio segment.

[0036] For ease of description, any one of the multiple audio segments is referred to as a first audio segment, and audio features of the first audio segment are obtained. The audio features may include at least one of a frequency feature, an amplitude feature, and an audio semantic feature, which is not limited here.

[0037] Step 104: Select a target image matching the first audio segment from a pre-acquired image library according to the audio features of the first audio segment.

[0038] The image library includes multiple images, and image features can be extracted for each image in advance. The image features can include at least one of pixel features and image semantic features. Pixel features are used to characterize the pixel-level features of an image. For example, at least one of the brightness and color of other pixels around each pixel is counted to characterize the pixel features of the pixel using the statistical results, or a convolutional neural network (CNN) is used to extract the pixel features of an image through multiple convolutional layers and pooling layers.

[0039] For example, an image is input into a multimodal large model to obtain the image semantic text information of the image, and the image semantic text information is input into a bidirectional encoder representation (Bidirectional Encoder Representations from Transformers, BERT) model to obtain the image semantic features of the image. A multimodal large model is a model that combines multimodal information such as text, images, videos, and audio for training.

[0040] It should be noted that various features in the present application are expressed in the form of vectors, and operations between features are operations between vectors.

[0041] The audio feature of the first audio segment is matched with the image feature of each image in the image library, for example, the cosine similarity is calculated between the audio feature of the first audio segment and the image feature of each image in the image library to obtain a similarity, which is a matching value. The target image to which the first audio segment matches is determined according to the matching value.

[0042] The target image may include one image or multiple images. It should be noted that the number of images included in the target image may be determined by user input and is not limited here.

[0043] After step 103 and step 104, each audio segment has a matching target image.

[0044] Step 105: synthesize the target image matched based on each of the audio segments with the initial audio to obtain a target video.

[0045] It should be noted that the target image matched by the audio segment is also described below as the target image corresponding to the audio segment.

[0046] The target image corresponding to each audio segment is synthesized with the initial audio. For example, the display order of the target image matched by each audio segment in the target video is set according to the playback order of each audio segment in the initial audio, so that the display order of the target image matched by the audio segment is consistent with the playback order of each audio segment, and the display duration of the target image matched by each audio segment is set according to the playback duration of each audio segment. It should be noted that the playback order of each audio segment in the initial audio is also the playback order of each audio segment in the target video.

[0047] In this embodiment, initial audio is obtained; the initial audio is segmented to obtain multiple audio segments; the multiple audio segments are traversed, and for any first audio segment among the multiple audio segments, the audio features of the first audio segment are obtained; based on the audio features of the first audio segment, a target image matching the first audio segment is selected from a pre-acquired image library; based on the target image matching each audio segment, the target image matching each audio segment is synthesized with the initial audio to obtain a target video. In the above, the target image matching each audio segment is determined by matching the audio features of each audio segment with the image features of the image in the image library, which improves the correlation between the audio segment and the image, and can make the audio and image in the generated target video more adaptable, thereby improving the quality of the target video.

[0048] In an embodiment of the present application, obtaining the audio feature of the first audio segment includes:

[0049] extracting frequency characteristics and amplitude characteristics of the first audio band;

[0050] Based on the frequency characteristics and amplitude characteristics of the first audio segment, performing principal component analysis on the first audio segment to obtain principal component characteristics of the first audio segment;

[0051] Extracting semantic features from the first audio segment to obtain audio semantic features of the first audio segment;

[0052] The principal component features and the audio semantic features of the first audio segment are fused to obtain audio features of the first audio segment.

[0053] In this embodiment, the frequency characteristics of the first audio segment can be extracted by Fast Fourier Transform (FFT). For example, the first audio segment is converted from the time domain to the frequency domain to obtain the energy distribution of the first audio segment at different frequencies, and then the energy distribution is analyzed to obtain the frequency characteristics of the first audio segment. The frequency characteristics may include information such as the main frequency and harmonics.

[0054] The amplitude of the first audio segment is analyzed, and volume fluctuation information is obtained according to the amplitude change. For example, the root mean square (RMS) can be used to calculate the square root of the average value of the instantaneous power of multiple sampling points in each sub-segment of the first audio segment. This value can reflect the average energy level of the sub-segment, that is, the amplitude. Based on the amplitude of each sub-segment, the amplitude feature of the first audio segment is constructed. For example, the amplitude feature of the first audio segment is composed of the amplitude of each sub-segment.

[0055] The main component of the first audio segment may be determined according to the frequency feature and the amplitude feature of the first audio segment to obtain the main component feature of the first audio segment.

[0056] The first audio segment may be processed using automatic speech recognition (ASR) technology to obtain audio semantic text information of the first audio segment, and the audio semantic text information may be input into a BERT model to obtain audio semantic features of the first audio segment.

[0057] When the principal component features and the audio semantic features of the first audio segment are fused, a weighted summation method can be used for the fusion. For example, a weight of 0.2 is set for the principal component features, and a weight of 0.8 is set for the audio semantic features. After weighted summation of the two, the audio features of the first audio segment are obtained.

[0058] It should be noted that the principal component features and the audio semantic features of the first audio segment have the same dimension, for example, both are 768 dimensions.

[0059] In this embodiment, frequency features and amplitude features are extracted from the first audio segment, and principal component analysis is performed to obtain principal component features of the first audio segment. The principal component features of the first audio segment are then fused with audio semantic features of the first audio segment to obtain audio features of the first audio segment. In this way, the audio features of the first audio segment fuse the frequency features, amplitude features, and audio semantic features of the first audio segment. That is, the audio features of the first audio segment contain information of multiple dimensions of the first audio segment, which can make the audio features of the first audio segment more accurately express the information of the first audio segment, thereby improving the subsequent adaptability of the first audio segment to the target image.

[0060] In the above, based on the frequency characteristics and amplitude characteristics of the first audio segment, principal component analysis is performed on the first audio segment to obtain the principal component characteristics of the first audio segment, including:

[0061] Obtaining a covariance matrix according to the frequency characteristics and amplitude characteristics of the first audio band;

[0062] Performing eigenvalue decomposition on the covariance matrix to obtain a plurality of eigenvalues ​​and an eigenvector corresponding to each eigenvalue;

[0063] Sort the plurality of eigenvalues ​​from largest to smallest to obtain the top N eigenvalues, where N is a positive integer;

[0064] The Mel-frequency spectrum feature of the first audio segment is multiplied by the feature vectors corresponding to the N eigenvalues ​​to obtain the principal component feature of the first audio segment.

[0065] In this embodiment, the Mel spectrum feature of the first audio segment can be subjected to principal component analysis, and a feature vector of a specified dimension can be generated using principal component analysis (PCA). Through PCA, a high-dimensional feature vector can be projected into a low-dimensional space, while removing noise and redundant information. Mel spectrum feature (Melspectrogram) is a frequency domain representation method that is more in line with the auditory characteristics of the human ear. It uses the Mel scale to simulate the difference in the human ear's perception of different frequencies. The Mel scale is a nonlinear frequency scale. Its basic principle is that the human ear is more sensitive to changes in lower frequencies and relatively insensitive to changes in high frequencies.

[0066] The data in the frequency feature and amplitude feature are standardized respectively so that the mean of each feature is 0 and the variance is 1, so as to ensure that different features are on the same scale and avoid some features dominating in PCA due to their large values.

[0067] According to the frequency feature data and amplitude feature data obtained after the above standardization, a covariance matrix is ​​constructed, and the covariance matrix reflects the correlation between different features. The covariance matrix is ​​subjected to eigenvalue decomposition to obtain multiple eigenvalues ​​and eigenvectors corresponding to each eigenvalue. The multiple eigenvalues ​​are sorted from large to small to obtain the top N eigenvalues, for example, N can be 768, and N can also select other values ​​according to actual conditions, which are not limited here.

[0068] The first audio segment is projected onto the principal component, specifically, the Mel frequency spectrum feature of the first audio segment is multiplied by a matrix consisting of eigenvectors corresponding to N eigenvalues ​​to obtain a reduced-dimensional eigenvector, which is the principal component feature of the first audio segment.

[0069] In this embodiment, a dominant principal component is determined according to the correlation between the frequency feature and the amplitude feature of the first audio segment, and then the first audio segment is projected onto the principal component, that is, the Mel frequency spectrum feature of the first audio segment is multiplied by the feature vector corresponding to the N eigenvalues ​​to obtain the principal component feature of the first audio segment, thereby realizing dimensionality reduction processing of the first audio segment. On the one hand, redundant information in the Mel frequency spectrum feature of the first audio segment is reduced to reduce the computational complexity. On the other hand, the principal component feature of the first audio segment is aligned with the audio semantic feature of the first audio segment in terms of dimension, so as to prepare for subsequent image matching calculation.

[0070] In yet another embodiment of the present application, before matching the audio feature of the first audio segment with the image feature of each image in the pre-acquired image library to obtain a plurality of matching values, the method further includes:

[0071] Performing pixel feature extraction and semantic feature extraction on each of the images in the image library to obtain pixel features and image semantic features of each image;

[0072] The pixel features and image semantic features of each of the images are fused to obtain image features of each of the images.

[0073] Specifically, image features may include at least one of pixel features and image semantic features. Pixel features are used to characterize pixel-level features of an image. For example, at least one of the brightness and color of other pixels around a certain pixel is counted, and the pixel features of the pixel are characterized by the statistical result. Alternatively, CNN is used to extract at least one feature of the brightness and color of the image through multiple convolutional layers and pooling layers, and multiple dimensions of features are generated through a fully connected layer, for example, 768 dimensions. It should be noted that if the brightness and color features of the image are extracted, the pixel features of the image can be obtained by weighted summing of the two features.

[0074] The image is input into the multimodal large model to obtain the image semantic text information of the image, and the image semantic text information is input into the BERT model to obtain the image semantic features of the image.

[0075] When fusing the pixel features and image semantic features of each image, a weighted summation method can be used for fusion. For example, a weight of 0.8 is set for the pixel features, and a weight of 0.2 is set for the image semantic features. After weighted summation of the two, the image features of the image are obtained.

[0076] It should be noted that the pixel features and image semantic features of an image have the same dimension, for example, both are 768 dimensions.

[0077] In this embodiment, pixel features and image semantic features of each image in the image library are extracted respectively, and pixel features and image semantic features of each image are obtained. Then, the pixel features and image semantic features of the image are fused to obtain image features of the image. In this way, the image features of the image fuse the pixel features and image semantic features of the image. That is to say, the image features of the image contain information of multiple dimensions of the image, which can make the image features of the image express the information of the image more accurately, thereby improving the adaptability of subsequent audio segments to the target image.

[0078] In yet another embodiment of the present application, selecting a target image matching the first audio segment from a pre-acquired image library according to the audio feature of the first audio segment includes:

[0079] Matching the audio features of the first audio segment with image features of each image in a pre-acquired image library to obtain a plurality of matching values;

[0080] A target image matching the first audio segment is selected from the image library according to the multiple matching values.

[0081] Specifically, the target image may include an image. For example, the image corresponding to the largest matching value may be selected as the target image of the first audio segment. Alternatively, a preset threshold is set and an image is randomly selected from images whose matching values ​​are greater than the preset threshold as the target image of the first audio segment.

[0082] The target image may also include multiple images. For example, multiple matching values ​​are sorted from large to small, and two or more images in the top order are selected as the target images of the first audio segment. Alternatively, a preset threshold is set, and two or more images are randomly selected from images with matching values ​​greater than the preset threshold as the target images of the first audio segment. The preset threshold can be set according to actual conditions and is not limited here.

[0083] In this embodiment, by sorting the multiple matching values, a target image that best matches the first audio segment can be selected, thereby improving the adaptability of the audio segment and the target image, thereby improving the quality of the target video.

[0084] In an embodiment of the present application, selecting a target image matching the first audio segment from the image library according to the multiple matching values ​​includes:

[0085] If the largest matching value among the plurality of matching values ​​is smaller than a preset threshold, taking the image corresponding to the largest matching value as the target image matching the first audio segment;

[0086] If there are M matching values ​​among the multiple matching values ​​that are greater than a preset threshold, U images are selected from the images corresponding to the M matching values ​​as target images matching the first audio segment, where M and U are both positive integers, and M is greater than or equal to U.

[0087] In the above, the preset threshold may be preset, for example, set to 0.8 or 0.9, which is not limited here. The target image may include one image or multiple images.

[0088] The audio features of the first audio segment and the image features of each image in the image library are calculated by cosine similarity to obtain similarity, which is the matching value. If the largest matching value among the multiple matching values ​​is less than the preset threshold, in this case, it means that there is no matching value greater than the preset threshold among the multiple matching values, then the image corresponding to the largest matching value is used as the target image corresponding to the first audio segment, so as to select an image that best matches the first audio segment and improve the adaptability of the first audio segment and the target image as much as possible; if there are M matching values ​​greater than the preset threshold among the multiple matching values, then U images are selected from the images corresponding to the M matching values ​​as the target images corresponding to the first audio segment, for example, the M matching values ​​are sorted from large to small, and the images corresponding to the U matching values ​​in the front sorting are selected as the target images corresponding to the first audio segment, so that the adaptability of the first audio segment and the target image is higher, or U images are randomly selected from the images corresponding to the M matching values ​​as the target images corresponding to the first audio segment, so that for the same first audio segment, different target images may be obtained due to randomness, which increases the fun.

[0089] In another embodiment of the present application, synthesizing the target image matched with each audio segment with the initial audio to obtain the target video includes:

[0090] According to the playback order of each audio segment in the initial audio, the display order of the target image matched by each audio segment in the target video is set, and according to the playback duration of each audio segment, the display duration of the target image matched by each audio segment is set;

[0091] Traversing the plurality of audio segments, for any second audio segment among the plurality of audio segments, performing weighted sum processing on an audio feature of the second audio segment and an image feature of a target image matched by the second audio segment, to obtain a target feature of the second audio segment;

[0092] According to the length of the target feature of the second audio segment, the display mode of the target image matched by the second audio segment is determined to obtain the target video. The display mode may refer to the switching effect of the target images corresponding to adjacent audio segments when switching displays.

[0093] Specifically, when performing weighted sum processing on the audio features of the second audio segment and the image features of the target image corresponding to the second audio segment, the weights of the audio features and the image features can be set according to actual conditions, and are not limited here. It should be noted that if the target image includes multiple images, the image participating in the weighted sum processing can be the last image displayed among the multiple images.

[0094] For example, if the target image includes an image, the display duration of the target image can be equal to the playback duration of the second audio segment, or slightly less than the playback duration of the second audio segment, and the remaining time can display the switching effect of switching to the target image corresponding to the adjacent audio segment.

[0095] For another example, if the target image includes multiple images, these multiple images can be displayed sequentially within the playback duration of the second audio segment. The display order of these multiple images can be random, or they can be displayed sequentially in order from large to small matching values, or they can be displayed sequentially in order from small to large matching values. These multiple images evenly divide the playback duration of the second audio segment.

[0096] When the length of the target feature is less than 1, a fade-in and fade-out method is used to gradually change the transparency of the image to achieve a smooth transition from the last target image displayed in the current audio segment to the first target image displayed in the adjacent audio segment. This is simple and natural. When the length of the target feature is in the range of 1-10, a dissolving method is used to mix the pixel values ​​of the last target image displayed in the current audio segment with the first target image displayed in the adjacent audio segment, gradually transitioning from one image to another, which can produce a soft transition effect. When the length of the target feature is greater than 10, a sliding method is used. The last target image displayed in the current audio segment slides out of the screen in a certain direction and speed, and the first target image displayed in the adjacent audio segment slides in to achieve a transition, which can increase the sense of dynamics.

[0097] In this embodiment, the display mode of the target image corresponding to the second audio segment in the target video is determined according to the length of the target feature. On the one hand, the display mode of the target image corresponding to the second audio segment is not limited to the pre-set mode, so that the display mode of the image is more flexible and more interesting. On the other hand, the target video can be made more visually smooth and natural, avoiding the abruptness of the image corresponding to the audio segment when switching. Since the display mode can be determined according to the audio segment, it can be dynamically adjusted in real time as the audio segment changes, thereby improving the quality and viewing experience of the target video, thereby improving the playback quality of the target video.

[0098] The following is an example of the video generation method provided in the embodiment of the present application. Figure 2 is another flow chart of the video generation method provided in the embodiment of the present application. Figure 2 As shown, the video generation method provided in the embodiment of the present application includes the following process:

[0099] Extracting audio semantic features and principal component features from the audio segment, and fusing the two to obtain the audio features of the audio segment;

[0100] Extract image semantic features and pixel features from the image, and fuse the two to obtain image features;

[0101] Calculate the similarity between the audio features of the audio segment and the image features of the image to obtain a target image matching the audio segment, thereby determining the corresponding relationship between the audio segment and the target image;

[0102] The audio features of the audio segment and the image features of the target image are fused to obtain a new feature vector, which is called the target feature. According to the length of the target feature, the target image of each audio segment is dynamically fused to obtain the target video.

[0103] The above video generation method is applied to a video generation module, wherein the video generation module includes: an audio input and preprocessing module, an audio feature extraction and analysis module, a digital image library and screening and adaptation module, a video synthesis and dynamic generation module, a real-time optimization and adjustment module, and a video output module. Each module is introduced as follows.

[0104] (I) Audio input and preprocessing module:

[0105] Supports input of multiple common audio formats, such as MP3, WAV, FLAC, etc.

[0106] Audio denoising: Use wavelet transform to remove background noise from audio and improve audio quality. Decompose the audio signal into wavelet coefficients of different frequencies and time scales. Noise is usually more obvious in the high-frequency part. By setting a threshold, the smaller wavelet coefficients (which may be noise) are set to zero, and then the inverse wavelet transform is performed to reconstruct the denoised audio. The wavelet transform method can effectively remove noise while retaining audio details, and has good processing capabilities for non-stationary signals.

[0107] Audio segmentation: According to the pause points of the audio, the audio (i.e. the initial audio obtained after denoising) is divided into several segments to obtain audio segments for subsequent more accurate feature extraction and processing.

[0108] It should be noted that the following feature extraction and image matching processes are all performed on audio segments as processing objects to obtain a target image that matches each audio segment. The following takes the first audio segment as an example to illustrate the feature extraction and image matching process.

[0109] (II) Audio feature extraction and semantic feature extraction module:

[0110] 1. Audio feature extraction:

[0111] (1) Frequency feature extraction: The frequency distribution features of the first audio segment, including the main frequency and harmonics, are extracted using the fast Fourier transform (FFT) method. The energy distribution of the first audio segment at different frequencies is obtained by converting the first audio segment from the time domain to the frequency domain. Then, the FFT results are analyzed to obtain information such as the main frequency and harmonics of the first audio segment.

[0112] (2) Amplitude feature extraction: Analyze the amplitude change of the first audio segment to obtain volume fluctuation information. For example, the root mean square (RMS) is used to calculate the square root of the average value of the instantaneous power of the first audio segment, reflecting the average energy level of the first audio segment, that is, the amplitude.

[0113] Use principal component analysis (PCA) to generate the specified 768-dimensional vector features. PCA can find the main components in the data, that is, those directions that can preserve the data variance to the greatest extent. Through PCA, high-dimensional feature vectors can be projected into a low-dimensional space while removing noise and redundant information. The specific operation is as follows: The data in the above frequency features and amplitude features are standardized so that the mean of each feature is 0 and the variance is 1. This ensures that different features are on the same scale and avoids some features from dominating the PCA due to their large values.

[0114] Calculate the covariance matrix of the standardized features. The covariance matrix reflects the correlation between different features. Perform eigenvalue decomposition on the covariance matrix to obtain eigenvalues ​​and eigenvectors. According to the size of the eigenvalue, select the eigenvectors corresponding to the first 768 largest eigenvalues. Project the first audio segment onto the selected principal component to obtain a new eigenvector. Perform matrix multiplication on the Mel spectrum feature of the first audio segment and the selected 768 eigenvectors to obtain the eigenvector after dimensionality reduction, that is, the principal component feature of the first audio segment. This feature is involved in subsequent operations and its dimension is 768.

[0115] 2. Semantic feature extraction: ASR technology is used to extract audio semantic features from the first audio segment. For example, ASR is used to perform signal processing on the first audio segment, split it into frames (millisecond level), and the split small waveforms are converted into multi-dimensional vector information according to human ear characteristics. These frame information are then identified into states and combined to form phonemes, and the phonemes are combined into words and strung into sentences to achieve semantic text extraction of audio.

[0116] The generated audio semantic text is transferred through the BERT model to obtain audio semantic features with a dimension of 768.

[0117] (III) Digital image library and screening adaptation module:

[0118] Build a massive digital image library of images of all kinds of subjects, styles, and scenes.

[0119] 1. Pixel feature extraction: Use commonly used convolutional layer image classification algorithms such as CNN to classify and annotate images in the image library, including themes, emotional colors, color features, etc. CNN extracts image features through multiple convolutional layers and pooling layers, and then classifies through a fully connected layer to generate corresponding 768-dimensional features.

[0120] 2. Image semantic feature extraction: A multimodal large model is used to extract the semantic text of the image. Specifically, it uses Vision Encoder to convert the image into a feature map, and aligns the image features with the text features through the interpolation layer. Then, the processed image features (image token embedding) and text features (texttoken embedding) are merged as the input of the language model to generate descriptive text and realize semantic text extraction of the image.

[0121] Finally, the semantic text of the generated image is transferred through the BERT model to obtain the image semantic features with a corresponding dimension of 768.

[0122] 3. Image screening based on audio features: The extracted 768-dimensional principal component features and audio semantic features are fused with a weight of 2:8 to obtain the audio features of the first audio segment; the 768-dimensional image semantic features and pixel features extracted from the image are fused with a weight of 2:8 to obtain the image features of the image. Then the cosine similarity of the audio features of the first audio segment and the image features of the image is calculated, and the image with the largest similarity value with the audio features of the first audio segment is screened from the image library.

[0123] Image adaptation: Perform operations such as resizing, color correction, and brightness adjustment on the selected images to make them better meet the needs of video generation.

[0124] (IV) Video synthesis and dynamic generation module:

[0125] The vector representation of the extracted audio features of the first frequency band and the vector representation of the pixel features of the target image matched with the first frequency band are weightedly combined to obtain the target features (hereinafter also referred to as the combined vector) to adopt different image transition effect designs. When the length of the combined vector is less than 1, a fade-in and fade-out method is adopted to gradually change the transparency of the image to achieve a smooth transition from one image to another, which is simple and natural. When the length of the combined vector is in the range of 1-10, a dissolving method is adopted to mix the pixel values ​​of the two images and gradually transition from one image to another, which can produce a soft transition effect. When the length of the combined vector is greater than 10, a sliding method is adopted, in which one image slides out of the screen in a certain direction and speed, and another image slides in at the same time to achieve transition, which can increase the sense of dynamics. Then set a suitable video frame rate to ensure the smoothness and stability of the video. Make the switching between images more natural and smooth.

[0126] (V) Real-time optimization and adjustment module:

[0127] Establish a multi-index evaluation system: including the matching degree between audio and image (for example, the matching degree is calculated by comparing the similarity between the audio features and the features of the selected image), the visual coherence of the video (analyzing whether the transition effect between adjacent images is natural and smooth, and whether the content and dynamic changes of the image are consistent with the coherence of visual perception. For example, it can be evaluated by calculating optical flow, analyzing image gradients, etc. to ensure that there are no abrupt visual changes in the video), image quality (for example, evaluating indicators such as image clarity, contrast, color saturation, and whether there are problems such as distortion and blur to ensure that the generated video has good image quality) and other indicators.

[0128] Real-time monitoring and feedback: During the target video generation process, the values ​​of the evaluation indicators are calculated in real time and the results are fed back to the system.

[0129] Feedback-based adjustment: According to feedback information, the image selection, transition effect, dynamic effect, etc. are adjusted and optimized in real time to improve the quality of the video. According to the gradient information of the evaluation index, the parameters such as image selection, transition effect, dynamic effect, etc. are adjusted to improve the evaluation index in the direction of optimization. It is widely used in parameter optimization and has a fast convergence speed.

[0130] (VI) Video output module:

[0131] Support output in multiple video formats, such as MP4, AVI, MOV, etc.

[0132] Video compression and encoding: Use efficient video compression algorithms, such as H.264 and H.265, to compress and encode the generated videos to reduce file size while ensuring video quality.

[0133] The video generation method in the embodiment of the present application can be applied to the following scenarios:

[0134] Multimedia creation: Provides efficient creation tools for music videos, animation production, etc., helping creators to quickly generate creative and artistic video works.

[0135] Advertising: Automatically generate attractive video content based on advertising audio to improve the communication effect and appeal of advertising.

[0136] Education and training: Generate videos by combining teaching audio and related images to enhance the interest and visualization of teaching content and improve learning outcomes.

[0137] Virtual Reality and Augmented Reality: Provide dynamic visual content for virtual scenes and augmented reality experiences to enhance user immersion.

[0138] The advantages of the video generation method in the embodiment of the present application are:

[0139] 1) Multi-dimensional audio feature extraction and analysis: Use relevant audio processing algorithms and ASR speech recognition technology to extract multi-dimensional features of the input audio, including semantic information and audio features (frequency, amplitude, rhythm, melody, etc.). Use the PCA principal component analysis algorithm to fuse relevant audio features and make relevant weighted connections with the semantic text features learned by the BERT model. Analyze and understand these features to accurately capture the emotions, themes, and content information expressed by the audio.

[0140] 2) Intelligent image screening and adaptation: A large-scale digital image library is established, and corresponding pixel features and image semantic features of the same dimension are generated through conventional CNN convolutional layer models and advanced multimodal large models (for example, Large Language and Vision Assistant (LLava), which is an end-to-end trained large multimodal model). Then, the images in the image library are matched by maximum cosine similarity based on audio features, and the digital images that are most relevant to the audio content and can best express its emotions are selected.

[0141] 3) Dynamic video generation and optimization: Using video synthesis technology, the screened digital images are dynamically combined and transitioned according to the length of the audio and image combined feature vector, making the generated video visually smoother and more natural, avoiding the abruptness when switching images, and being able to make real-time dynamic adjustments based on audio changes, thereby improving the quality and viewing experience of the video.

[0142] The video generation method of the embodiment of the present application can generate a target video based on only a piece of audio, which greatly improves the efficiency and automation of video generation, reduces manual participation, and reduces production costs. The generated video has a high degree of consistency and coherence with the audio in terms of content and emotional expression, which enhances the artistic value and viewing experience of the video. It can process audio of various types and styles and has wide applicability and versatility. It provides new tools and ideas for multimedia creation, advertising production, education and training and other fields.

[0143] Figure 3 FIG. 1 shows a structural diagram of a video generation device provided in an embodiment of the present application. Figure 3 As shown, the video generating device 300 includes:

[0144] A first acquisition module 301 is used to acquire initial audio;

[0145] A segmentation module 302 is used to segment the initial audio to obtain multiple audio segments;

[0146] A second acquisition module 303 is used to traverse the multiple audio segments, and for any first audio segment among the multiple audio segments, acquire an audio feature of the first audio segment;

[0147] A matching module 304, configured to select a target image matching the first audio segment from a pre-acquired image library according to the audio features of the first audio segment;

[0148] The synthesis module 305 is used to synthesize the target image matched with each audio segment with the initial audio to obtain a target video.

[0149] In an embodiment of the present application, the second acquisition module 303 includes:

[0150] A first feature extraction submodule, used to extract frequency features and amplitude features of the first audio segment;

[0151] an analysis submodule, configured to perform principal component analysis on the first audio segment based on the frequency characteristics and amplitude characteristics of the first audio segment to obtain principal component characteristics of the first audio segment;

[0152] A second feature extraction submodule, configured to extract semantic features from the first audio segment to obtain audio semantic features of the first audio segment;

[0153] The fusion submodule is used to fuse the principal component features and the audio semantic features of the first audio segment to obtain the audio features of the first audio segment.

[0154] In one embodiment of the present application, the analysis submodule includes:

[0155] A first acquisition unit, configured to obtain a covariance matrix according to frequency characteristics and amplitude characteristics of the first audio segment;

[0156] A decomposition unit, used for performing eigenvalue decomposition on the covariance matrix to obtain a plurality of eigenvalues ​​and an eigenvector corresponding to each eigenvalue;

[0157] A sorting unit, used for sorting the plurality of eigenvalues ​​from largest to smallest to obtain N eigenvalues ​​ranked first, where N is a positive integer;

[0158] The second acquisition unit is configured to multiply the Mel frequency spectrum feature of the first audio segment by the feature vectors corresponding to the N eigenvalues ​​to obtain the principal component feature of the first audio segment.

[0159] In an embodiment of the present application, the video generating device 300 further includes:

[0160] An extraction module, used for performing pixel feature extraction and semantic feature extraction on each of the images in the image library to obtain pixel features and image semantic features of each image;

[0161] The fusion module is used to fuse the pixel features and image semantic features of each of the images to obtain the image features of each of the images.

[0162] In one embodiment of the present application, the synthesis module 305 includes:

[0163] A submodule is set, which is used to set the display order of the target image matched by each audio segment in the target video according to the playback order of each audio segment in the initial audio, and to set the display duration of the target image matched by each audio segment according to the playback duration of each audio segment;

[0164] a traversal submodule, configured to traverse the plurality of audio segments, and for any second audio segment among the plurality of audio segments, perform weighted sum processing on an audio feature of the second audio segment and an image feature of a target image matched by the second audio segment, so as to obtain a target feature of the second audio segment;

[0165] The acquisition submodule is used to determine the display mode of the target image matched by the second audio segment according to the length of the target feature of the second audio segment, so as to obtain the target video.

[0166] In one embodiment of the present application, the matching module 304 includes:

[0167] a matching submodule, configured to match the audio features of the first audio segment with image features of each image in a pre-acquired image library to obtain a plurality of matching values;

[0168] The selection submodule is used to select a target image matching the first audio segment from the image library according to the multiple matching values.

[0169] In one embodiment of the present application, the selection submodule includes:

[0170] a first selection unit, configured to use an image corresponding to the largest matching value as a target image matching the first audio segment if the largest matching value among the plurality of matching values ​​is smaller than a preset threshold;

[0171] A second selection unit is configured to select U images from the images corresponding to the M matching values ​​as target images matching the first audio segment if M matching values ​​among the multiple matching values ​​are greater than a preset threshold, where M and U are both positive integers and M is greater than or equal to U.

[0172] The video generating device 300 provided in the embodiment of the present application can implement each process implemented in the aforementioned video generating method embodiment and achieve the same technical effect. To avoid repetition, it will not be described here.

[0173] Figure 4 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application is shown.

[0174] The electronic device may include a processor 401 and a memory 402 storing computer program instructions.

[0175] Specifically, the processor 401 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0176] Memory 402 may include a large capacity memory for data or instructions. By way of example and not limitation, memory 402 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, memory 402 may include a removable or non-removable (or fixed) medium. In appropriate cases, memory 402 may be inside or outside of an integrated gateway disaster recovery device. In a specific embodiment, memory 402 is a non-volatile solid-state memory.

[0177] The memory may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Thus, typically, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to the first aspect or the second aspect of the present disclosure.

[0178] The processor 401 implements any one of the information auditing methods in the above embodiments by reading and executing computer program instructions stored in the memory 402 .

[0179] In one example, the electronic device may further include a communication interface 403 and a bus 410. Figure 4 As shown, the processor 401, the memory 402, and the communication interface 403 are connected via a bus 410 and communicate with each other.

[0180] The communication interface 403 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0181] Bus 410 includes hardware, software or both, and the parts of information audit method or verification equipment are coupled to each other. For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 410 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the application considers any suitable bus or interconnection.

[0182] In addition, in combination with the video generation method in the above embodiment, the embodiment of the present application can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any one of the video generation methods in the above embodiment is implemented.

[0183] In addition, the embodiments of the present application may be implemented by providing a computer program product. When instructions in the computer program product are executed by a processor of an electronic device, the electronic device implements any one of the video generation methods in the above embodiments.

[0184] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0185] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0186] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0187] Aspects of the present disclosure are described above with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0188] The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. A video generation method, characterized in that: The method comprises: Get the initial audio; Segmenting the initial audio to obtain multiple audio segments; Traversing the plurality of audio segments, and for any first audio segment among the plurality of audio segments, acquiring an audio feature of the first audio segment; selecting, according to the audio features of the first audio segment, a target image matching the first audio segment from a pre-acquired image library; A target image matched based on each of the audio segments is synthesized with the initial audio to obtain a target video.

2. The method according to claim 1, characterized in that Acquiring the audio features of the first audio segment includes: extracting frequency characteristics and amplitude characteristics of the first audio band; Based on the frequency characteristics and amplitude characteristics of the first audio segment, performing principal component analysis on the first audio segment to obtain principal component characteristics of the first audio segment; Extracting semantic features from the first audio segment to obtain audio semantic features of the first audio segment; The principal component features and the audio semantic features of the first audio segment are fused to obtain audio features of the first audio segment.

3. The method according to claim 2, characterized in that Based on the frequency characteristics and amplitude characteristics of the first audio segment, principal component analysis is performed on the first audio segment to obtain principal component characteristics of the first audio segment, including: Obtaining a covariance matrix according to the frequency characteristics and amplitude characteristics of the first audio band; Performing eigenvalue decomposition on the covariance matrix to obtain a plurality of eigenvalues ​​and an eigenvector corresponding to each eigenvalue; Sort the plurality of eigenvalues ​​from largest to smallest to obtain the top N eigenvalues, where N is a positive integer; The Mel-frequency spectrum feature of the first audio segment is multiplied by the feature vectors corresponding to the N eigenvalues ​​to obtain the principal component feature of the first audio segment.

4. The method according to claim 1, characterized in that: Before matching the audio feature of the first audio segment with the image features of each image in the pre-acquired image library to obtain a plurality of matching values, the method further includes: Performing pixel feature extraction and semantic feature extraction on each of the images in the image library to obtain pixel features and image semantic features of each of the images; The pixel features and image semantic features of each of the images are fused to obtain image features of each of the images.

5. The method according to claim 1, characterized in that The target image matched with each audio segment is synthesized with the initial audio to obtain a target video, including: According to the playback order of each audio segment in the initial audio, the display order of the target image matched by each audio segment in the target video is set, and according to the playback duration of each audio segment, the display duration of the target image matched by each audio segment is set; Traversing the plurality of audio segments, for any second audio segment among the plurality of audio segments, performing weighted sum processing on an audio feature of the second audio segment and an image feature of a target image matched by the second audio segment, to obtain a target feature of the second audio segment; According to the length of the target feature of the second audio segment, a display mode of the target image matched by the second audio segment is determined to obtain a target video.

6. The method according to claim 1, characterized in that Selecting a target image matching the first audio segment from a pre-acquired image library according to the audio feature of the first audio segment includes: Matching the audio features of the first audio segment with image features of each image in a pre-acquired image library to obtain a plurality of matching values; A target image matching the first audio segment is selected from the image library according to the multiple matching values.

7. The method according to claim 6, characterized in that Selecting a target image matching the first audio segment from the image library according to the multiple matching values ​​includes: If the largest matching value among the plurality of matching values ​​is smaller than a preset threshold, taking the image corresponding to the largest matching value as the target image matching the first audio segment; If there are M matching values ​​among the multiple matching values ​​that are greater than a preset threshold, U images are selected from the images corresponding to the M matching values ​​as target images matching the first audio segment, where M and U are both positive integers, and M is greater than or equal to U.

8. A video generating device, characterized in that: The device comprises: A first acquisition module, used to acquire initial audio; A segmentation module, used to segment the initial audio to obtain multiple audio segments; A second acquisition module is used to traverse the multiple audio segments, and for any first audio segment among the multiple audio segments, acquire an audio feature of the first audio segment; a matching module, configured to select a target image matching the first audio segment from a pre-acquired image library according to the audio features of the first audio segment; A synthesis module is used to synthesize the target image matched with each audio segment with the initial audio to obtain a target video.

9. An electronic device, characterized in that: include: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the video generating method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the video generation method according to any one of claims 1 to 7 is implemented.

11. A computer program product, characterized in that When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the video generating method as described in any one of claims 1 to 7.

Images