A video generation method and computer device

Abstract

The present invention provides a method for generating a video and a computer device. The method for generating a video includes: obtaining an image set to be processed and an audio to be processed, wherein the image set includes a plurality of images; based on the image set and the audio, determining a target beat point corresponding to each of the plurality of images; inserting an image corresponding to the target beat point at each target beat point in the audio to generate a video. In the present invention, each image is inserted at a target beat point in the audio, which can improve the expressiveness of the video and obtain a video with better quality.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a video generation method and computer equipment. Background Art

[0002] Video production is a complex process. To get a high-quality video, you need to organize the images, select the appropriate audio, and then determine the insertion point of the image. This process has relatively high requirements on the producer's technical ability.

[0003] For ordinary people who do not have video production technology, they usually manually select images and add background audio to play them in the form of a slide show. The videos obtained in this way have poor expressiveness and low quality.

[0004] Therefore, the prior art needs to be improved. Summary of the invention

[0005] The present invention provides a video generation method and a computer device, so as to enable similar images to be played continuously, and each image is inserted at the beat point of the audio, so as to improve the expressiveness of the video and obtain a video with better quality.

[0006] In a first aspect, an embodiment of the present invention provides a method for generating a video, comprising:

[0007] Acquire an image set to be processed and an audio to be processed, wherein the image set includes a plurality of images;

[0008] Based on the image set and the audio, determining a target beat point corresponding to each of the plurality of images;

[0009] An image corresponding to each target beat point in the audio is inserted to generate a video.

[0010] In a further improved solution, determining the target beat point corresponding to each of the plurality of images based on the image set and the audio specifically includes:

[0011] Based on the similarity between any two images in the image set, the plurality of images are sorted to obtain an image insertion sequence;

[0012] Obtaining a number of beats of the audio;

[0013] The target beat point corresponding to each image in the image insertion sequence is determined according to the image insertion sequence, the multiple beat points and the audio, wherein the target beat point is a beat point among the multiple beat points of the audio for inserting the image.

[0014] In a further improved solution, the sorting of the plurality of images based on the similarity between any two images in the image set to obtain an image insertion sequence specifically includes:

[0015] Selecting a starting image from the image set, and setting the insertion sequence number of the starting image to the first sequence number;

[0016] Determine a non-starting image set corresponding to the starting image, wherein the non-starting image set includes a plurality of non-starting images;

[0017] Determine a candidate image corresponding to the starting image based on the similarity between each non-starting image and the starting image, and set the insertion sequence number of the candidate image to the next sequence number after the insertion sequence number of the starting image;

[0018] The candidate image is used as a starting image, and the step of determining a non-starting image set corresponding to the starting image is continued until insertion sequence numbers corresponding to all images in the image set are determined;

[0019] The image insertion sequence corresponding to the image set is determined according to the insertion sequence numbers corresponding to all the images in the image set.

[0020] In a further improved solution, determining the non-starting image set corresponding to the starting image specifically includes:

[0021] For the starting image, all images whose insertion sequence numbers have not been determined are selected from the image set to obtain a non-starting image set corresponding to the starting image.

[0022] In a further improved solution, determining the candidate image corresponding to the starting image based on the similarity between each non-starting image in the non-starting image set and the starting image specifically includes:

[0023] Calculating the similarity between each non-starting image in the non-starting image set and the starting image respectively to obtain a similarity set;

[0024] A maximum similarity is selected from the similarity set, and an image corresponding to the maximum similarity is used as a candidate image corresponding to the starting image.

[0025] In a further improved solution, determining the target beat point corresponding to each image in the image insertion sequence according to the image insertion sequence, the plurality of beat points and the audio specifically includes:

[0026] Obtaining the audio duration corresponding to the audio and the number of images of the plurality of images;

[0027] Determining the image insertion point corresponding to each image in the image insertion sequence according to the number of images and the audio duration, wherein the duration between two adjacent image insertion points is determined according to the audio duration and the number of images;

[0028] The target beat points corresponding to each of the plurality of images are determined according to the plurality of image insertion points and the plurality of beat points.

[0029] In a further improved solution, determining the target beat points corresponding to each of the plurality of images according to the plurality of image insertion points and the plurality of beat points specifically includes:

[0030] For each image insertion point, a beat point closest to the image insertion point is determined among the plurality of beat points, and the beat point closest to the image insertion point is used as a target beat point of the image corresponding to the image insertion point.

[0031] In a further improved solution, inserting an image corresponding to each target beat point in the audio to generate a video specifically includes:

[0032] For each target beat point, when the playing time of the audio reaches the target beat point, an image corresponding to the target beat point is inserted, and the image is used as an image frame played between the target beat point and the next target beat point.

[0033] In a further improved solution, before obtaining the image set to be processed and determining the image insertion sequence based on the similarity between any two images in the image set, the method further includes:

[0034] Acquire an original image set, wherein the original image set includes a plurality of original images, and the plurality of original images include at least one template image;

[0035] Determine and extract a target feature map corresponding to each original image in the original image set;

[0036] Dividing the original image set into original image subsets of different categories based on all determined target feature maps;

[0037] Any original image subset including the template image is used as the image set to be processed.

[0038] In a second aspect, the present invention provides a video generation device, comprising:

[0039] An acquisition unit, used to acquire an image set to be processed and an audio to be processed, wherein the image set includes a plurality of images;

[0040] A target beat point determination unit, configured to determine a target beat point corresponding to each of the plurality of images based on the image set and the audio;

[0041] The video generating unit is used to insert an image corresponding to each target beat point in the audio to generate a video.

[0042] In a third aspect, an embodiment of the present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0043] Acquire an image set to be processed and an audio to be processed, wherein the image set includes a plurality of images;

[0044] Based on the image set and the audio, determining a target beat point corresponding to each of the plurality of images;

[0045] An image corresponding to each target beat point in the audio is inserted to generate a video.

[0046] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0047] Acquire an image set to be processed and an audio to be processed, wherein the image set includes a plurality of images;

[0048] Based on the image set and the audio, determining a target beat point corresponding to each of the plurality of images;

[0049] An image corresponding to each target beat point in the audio is inserted to generate a video.

[0050] Compared with the prior art, the embodiments of the present invention have the following advantages:

[0051] The present invention provides a method for generating a video, comprising: obtaining an image set to be processed and an audio to be processed, wherein the image set includes a plurality of images; based on the image set and the audio, determining a target beat point corresponding to each of the plurality of images; and inserting an image corresponding to the target beat point at each target beat point in the audio to generate a video. In the present invention, each image is inserted at a beat point of the audio, which can improve the expressiveness of the video and obtain a video with better quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0053] Figure 1 A schematic diagram of a flow chart of a method for generating a video in an embodiment of the present invention;

[0054] Figure 2 Schematic diagram of a feature extraction network in an embodiment of the present invention;

[0055] Figure 3 is a structural schematic diagram of a video generating device in an embodiment of the present invention;

[0056] Figure 4 1 is a diagram showing the internal structure of a computer device in an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0058] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0059] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as herein.

[0060] After research, the inventor found that video production is a complicated process. To obtain a high-quality video, it is necessary to organize images, select appropriate audio, and then determine the insertion point of the image. This process has relatively high requirements on the technical ability of the producer. For ordinary people who do not have video production technology, they usually manually select images and add background audio to play in the form of a slide show. The video obtained in this way has poor expressiveness and low quality.

[0061] In order to solve the above problem, in an embodiment of the present invention, an image set to be processed and an audio to be processed are obtained, wherein the image set includes a plurality of images; based on the image set and the audio, a target beat point corresponding to each of the plurality of images is determined; and at each target beat point in the audio, an image corresponding to the target beat point is inserted to generate a video. In the present invention, each image is inserted at a beat point of the audio, which can improve the expressiveness of the video and obtain a video with better quality.

[0062] A video generation method provided in an embodiment of the present invention can be applied to electronic devices, and the electronic devices may include: a PC, a television, a server, a mobile phone, a tablet computer, a PDA, a personal digital assistant (PDA), etc.

[0063] The invention is further described below by describing embodiments in conjunction with the accompanying drawings.

[0064] See also Figure 1 , this embodiment provides a method for generating a video, including:

[0065] S1. Obtain an image set to be processed and an audio to be processed, wherein the image set includes a plurality of images.

[0066] In an embodiment of the present invention, the image set to be processed includes a plurality of images used to generate a video. The image set to be processed may be collected by a terminal that executes the method for generating the video, or the image set to be processed may be obtained from a third-party device, or some of the images in the image set to be processed are from a third-party device and some are collected by the terminal.

[0067] In the embodiment of the present invention, the audio is used to generate a video. In order to obtain a better video effect, the audio can be the audio corresponding to the music, and the audio is used as the background music of the video.

[0068] In an embodiment of the present invention, in order to make the images in the image set to be processed similar in style and obtain a video with more harmonious pictures, the images in the image set can be limited to belong to the same category. Specifically, images belonging to the same category can be selected in the original image set, and the selected images belonging to the same category are used as the image set to be processed.

[0069] Specifically, before step S1, the following steps are included:

[0070] M1. Obtain an original image set, wherein the original image set includes multiple original images, and the multiple original images include at least one template image.

[0071] In an embodiment of the present invention, the original image set includes multiple original images. Similarly, the original images can be obtained from the network, or the original images can be collected by the terminal. The multiple original images include at least one template image. When selecting an image for generating a video from multiple original images, the priority of the template image is higher than the priority of the non-template image. The non-template image refers to other original images in the multiple original images except the template image; that is, for one template image and one non-template image, the template image can be selected as the image for generating a video.

[0072] The template image may be a user-defined selection, that is, the user specifies the template image from multiple original images, and the template image may be the user's favorite original image, indicating that the user most wants the template image to appear in the video. There may be multiple template images.

[0073] M2. Determine the target feature map corresponding to each original image in the original image set.

[0074] In an embodiment of the present invention, a target feature map corresponding to each original image can be determined by a neural network model. The target feature map is a multi-channel image (only one pixel) with an image size of 1*1, and the target feature map can be represented by a vector, the dimension of the vector is the same as the number of channels of the target feature map, and the value of a dimension in the vector is the pixel value of the pixel in the channel corresponding to the dimension.

[0075] An original image is input into the neural network model, and the output of the neural network model is the target feature map corresponding to the original image. The neural network model includes a feature extraction module and a fully connected module; the original image is input into the feature extraction module to obtain the feature map corresponding to the original image, and the feature map is input into the fully connected module, and the output of the fully connected module is the target feature map.

[0076] In the embodiment of the present invention, the image size of the original image input to the neural network model needs to meet the input requirements of the neural network model, and the image size of all original images needs to be adjusted to a preset size in advance. For example, the preset size is 224*224.

[0077] In a specific implementation, the neural network model may be a VGG16 network model. The feature extraction network used to generate the feature atlas is as follows: Figure 2 As shown, the neural network model includes five feature extraction modules and one fully connected module, wherein the five feature extraction modules include: a first feature extraction module, a second feature extraction module, a third feature extraction module, a fourth feature extraction module and a fifth feature extraction module.

[0078] The first feature extraction module includes: a first convolution layer c1, a second convolution layer c2 and a first pooling layer p1; the sizes of the convolution kernels of c1 and c2 are both 3*3*3, the numbers of the convolution kernels of c1 and c2 are both 64, and the parameter of p1 is 2*2; the input item of the first feature extraction module is an original image, and the first feature extraction module extracts the features of the original image to obtain a first feature map.

[0079] The second feature extraction module includes: a third convolution layer c3, a fourth convolution layer c4 and a second pooling layer p2; the sizes of the convolution kernels of c3 and c4 are both 3*3*3, the numbers of the convolution kernels of c3 and c4 are both 128, and the parameter of p2 is 2*2; the input item of the second feature extraction module is the first feature map, and the second feature extraction module extracts the features of the first feature map to obtain the second feature map.

[0080] The third feature extraction module includes: a fifth convolutional layer c5, a sixth convolutional layer c6, a seventh convolutional layer c7 and a third pooling layer p3; the sizes of the convolution kernels of c5, c6 and c7 are all 3*3*3, the numbers of the convolution kernels of c5, c6 and c7 are all 256, and the parameter of p3 is 2*2; the input item of the third feature extraction module is the second feature map, and the third feature extraction module extracts the features of the second feature map to obtain the third feature map.

[0081] The fourth feature extraction module includes: an eighth convolution layer c8, a ninth convolution layer c9, a tenth convolution layer c10 and a fourth pooling layer p4; the sizes of the convolution kernels of c8, c9 and c10 are all 3*3*3, the numbers of the convolution kernels of c8, c9 and c10 are all 512, and the parameter of p4 is 2*2; the input item of the fourth feature extraction module is the third feature map, and the fourth feature extraction module extracts the features of the third feature map to obtain a fourth feature map.

[0082] The fifth feature extraction module includes: an eleventh convolutional layer c11, a twelfth convolutional layer c12, a thirteenth convolutional layer c13 and a fifth pooling layer p5; the sizes of the convolution kernels of c10, c11 and c12 are all 3*3*3, the numbers of the convolution kernels of c11, c12 and c13 are all 512, and the parameter of p5 is 2*2; the input item of the fifth feature extraction module is the fourth feature map, and the fifth feature extraction module extracts the features of the fourth feature map to obtain the fifth feature map.

[0083] The fully connected module includes a first fully connected layer fc1, a second fully connected layer fc2 and a third fully connected layer fc3. The parameters of the first fully connected layer are: 1*1*4096, the parameters of the second fully connected layer are 1*1*4096, and the parameters of the third fully connected layer are 1*1*1000. The fifth feature map is input into the fully connected module to obtain a target feature map corresponding to the original image. The target feature map can be represented in a vector form, for example, the target feature map can be represented as: {x1, x2, ..., xn}.

[0084] M3. Divide the original image set into original image subsets of different categories based on all determined target feature maps.

[0085] In the embodiment of the present invention, a classification method is used to divide the original image set into multiple original image subsets, and the categories corresponding to each original image subset are different. For example, the original image set can be divided into original image subsets of different categories by a K-means algorithm. The styles of the original images included in an original image subset are similar.

[0086] Specifically, k target feature maps are randomly selected from all target feature maps as initial centroids. For example, if k is set to 3, the initial centroids can be obtained as u1, u2 and u3. All target feature maps except the initial centroids are recorded as feature maps to be classified. For each feature map to be classified, the distance between the feature map to be classified and each initial centroid: u1, u2 and u3 is calculated respectively. The initial centroid corresponding to the minimum distance and the feature map to be classified are classified into the same category to obtain several classification sets.

[0087] For example, there are 10 target feature maps, including: t1, t2, ..., t10. Three target feature maps are randomly selected from the 10 target feature maps: t1, t2 and t3. T1 is recorded as the initial centroid u1, t2 is recorded as the initial centroid u2, and t3 is recorded as the initial centroid u3. The feature maps to be classified include: t4, t5, ..., t10. For each feature map to be classified, the distance between the feature map to be classified and each initial centroid is calculated. For example, for t4, the distance between t4 and u1 is calculated to obtain d41, the distance between t4 and u2 is calculated to obtain d42, and the distance between t4 and u3 is calculated to obtain d43. Assuming that d43 is the smallest, the original image corresponding to t4 and the original image corresponding to u3 are divided into one category. Perform the above calculations on all feature maps to be classified to obtain 3 classification sets.

[0088] In an embodiment of the present invention, for each classification set in a plurality of classification sets, the classification centroid corresponding to the classification set is determined, and for each feature graph to be classified in the above-mentioned feature graphs to be classified, the distance between the feature graph to be classified and each classification centroid is calculated, and the classification centroid corresponding to the minimum distance and the feature graph to be classified are classified into the same category to obtain a plurality of updated classification sets.

[0089] The classification centroid corresponding to the classification set can be determined by formula (1).

[0090]

[0091] Among them, Cj is the classification set, t is the target feature map belonging to the classification set Cj, and uj is the classification centroid.

[0092] In an embodiment of the present invention, the following steps are repeated: calculating the classification corresponding to the classification set: "for each classification set in the plurality of classification sets, determining the classification centroid corresponding to the classification set" until the classification centroid is the same as the classification centroid calculated last time, and then treating the plurality of classification sets as subsets of the original image of different categories.

[0093] M4. Use any original image subset including the template image as the image set to be processed.

[0094] In an embodiment of the present invention, an original image subset including a template image is determined from among several original image subsets. If the number of original image subsets including the template image is greater than 1, one of the multiple original image subsets including the template image is randomly determined as the image set to be processed. Alternatively, the multiple template images have their own corresponding likenesses, which can be set by the user, and the original image subset including the template image with the highest likeness is used as the image set to be processed. Alternatively, the likeness and the number of original images in the original image subset are combined to determine the image set to be processed from among several original image subsets including the template image.

[0095] S2. Determine a target beat point corresponding to each of the plurality of images based on the image set and the audio.

[0096] In the embodiment of the present invention, the signal corresponding to the audio is analyzed to determine several beat points corresponding to the audio, and the beat point is the moment when a musical instrument plays a specific note, such as a drum beat. The audio includes a large number of beat points, and any two different beat points correspond to different playback times. The target beat point is actually a beat point in the audio, and the target beat point is a beat point used to insert an image among several beat points of the audio.

[0097] In the embodiment of the present invention, the playback order of the plurality of images is first determined, and then the target beat point of the image is determined according to the playback order of the plurality of images. The playback order of the images in the video can be determined based on the similarity between any two images in the plurality of images, and similar images can be set to be played continuously, so that when the video is played, the style of the adjacently played images will not be too jumpy.

[0098] Specifically, step S2 includes:

[0099] S21. Sort the images based on the similarity between any two images in the image set to obtain an image insertion sequence.

[0100] In an embodiment of the present invention, the image insertion sequence includes a plurality of images and an insertion sequence number corresponding to each image, and the plurality of images in the image insertion sequence can be arranged according to the insertion sequence number. When generating a video, the plurality of images are inserted into the audio in order from small to large insertion sequence numbers.

[0101] In an embodiment of the present invention, for two adjacent images in the image insertion sequence, the two images are respectively a first image and a second image, wherein the first image is arranged before the second image, then the similarity between the first image and the second image is greater than the similarity between any image arranged after the second image and the first image.

[0102] Specifically, step S21 includes:

[0103] S211, selecting a starting image from the image set, and setting the insertion sequence number of the starting image to the first sequence number.

[0104] In the embodiment of the present invention, the starting image may be randomly selected, and the insertion sequence number corresponding to the starting image is set as the first sequence number, and the first sequence number may be represented by a number, for example, the number 1 represents the first sequence number.

[0105] S212: Determine a non-starting image set corresponding to the starting image, wherein the non-starting image set includes several non-starting images.

[0106] In the embodiment of the present invention, for the starting image, all images whose insertion numbers have not been determined are selected from the image set, and the images whose insertion numbers have not been determined are used as non-starting images, and a non-starting image set corresponding to the starting image is obtained based on all non-starting images. That is, the non-starting image set includes a number of non-starting images, and the non-starting images are images for which insertion numbers have not been set.

[0107] For example, the image set includes images: r1, r2, r3, ..., r8. In the above step, the insertion sequence number of r1 has been determined to be the first sequence number, and the insertion sequence numbers of r2, r3, ..., r8 have not been determined, so r2, r3, ..., r8 are non-starting images, and the non-starting image set is: {r2, r3, ..., r8}.

[0108] S213: Determine a candidate image corresponding to the starting image based on the similarity between each non-starting image and the starting image, and set the insertion sequence number of the candidate image to be the next sequence number after the insertion sequence number of the starting image.

[0109] In the embodiment of the present invention, "setting the insertion sequence number of the candidate image to the next sequence number after the insertion sequence number of the starting image" means that the candidate image is located next to the starting image in the image insertion sequence. For each non-starting image, the similarity between the non-starting image and the starting image is calculated.

[0110] Specifically, step S213 includes:

[0111] S2131 , respectively calculating the similarity between each non-starting image in the non-starting image set and the starting image to obtain a similarity set.

[0112] In an embodiment of the present invention, for each non-starting image, a target feature map corresponding to the non-starting image and a target feature map of the starting image are obtained, and the similarity between each non-starting image and the starting image is calculated to obtain the similarity corresponding to each non-starting image, thereby obtaining a similarity set.

[0113] Specifically, the similarity between the non-starting image and the starting image can be calculated by formula (2).

[0114]

[0115] Among them, the characteristics of the starting image rx are tx = {x1, x2, ..., xn}, the characteristics of the non-starting image ry are ty = {y1, y2, ...yn}, and SIM(x, y) is the similarity between the starting image rx and the non-starting image ry.

[0116] S2132: Select a maximum similarity in the similarity set, and use an image corresponding to the maximum similarity as a candidate image corresponding to the starting image.

[0117] In the embodiment of the present invention, the similarity is represented by a numerical value. The smaller the numerical value, the smaller the similarity, and the larger the numerical value, the greater the similarity. The maximum similarity refers to the similarity with the largest numerical value among all similarities. The image corresponding to the maximum similarity is used as the candidate image corresponding to the starting image.

[0118] For example, r1 is the starting image, r2, r3, …, r8 are non-starting images, where the similarity between r3 and r1 is the maximum similarity in the similarity set, then r3 is taken as the candidate image of r1, and the insertion sequence number corresponding to r3 is the next sequence number after the insertion sequence number of r1. Assuming that the insertion sequence number corresponding to r1 is 1, indicating that the first inserted image is r1, then the insertion sequence number corresponding to r3 is 2, indicating that the second inserted image is r3, where the insertion sequence number "2" is the next to the insertion sequence number "1". In the image insertion sequence, image r3 is arranged after image r1.

[0119] S214: Use the candidate image as the starting image, and continue to perform the step of determining the non-starting image set corresponding to the starting image until the insertion sequence numbers corresponding to all the images in the image set are determined.

[0120] In the embodiment of the present invention, after steps S211 to S213, only the insertion numbers of two images are determined, and the insertion numbers of other images in the image set to be processed need to be determined. For the sake of convenience, the candidate image is taken as the starting image, and step S212 is continued to be executed to determine the candidate image corresponding to the starting image.

[0121] For example, after step S211 to step S123, it is determined that the insertion number of r1 is 1 and the insertion number of r3 is 2. Next, it is necessary to determine the candidate image corresponding to r3, that is, the next image of r3 during playback. Take r3 as the starting image, first determine the non-starting image corresponding to r3 (starting image). It has been explained above that the non-starting image is all images whose insertion numbers have not been determined. In this example, since the insertion numbers of r1 and r3 have been determined, the non-starting images corresponding to r3 include: r2, r4, r5, ..., r8. Calculate the similarity between r3 and each non-starting image (r2, r4, r5, ..., r8) respectively, and determine the candidate image corresponding to r3. Assuming that the candidate image corresponding to r3 is r7, the insertion number of r7 is the next one of the insertion number corresponding to r3. In the image insertion sequence, r7 is arranged next to r3. Execute step S212 again to determine the candidate image corresponding to r7 until the insertion numbers corresponding to all images in the image set are determined.

[0122] S215 . Determine an image insertion sequence corresponding to the image set according to the insertion sequence numbers corresponding to all the images in the image set.

[0123] In the embodiment of the present invention, after the insertion sequence numbers corresponding to all the images are determined, all the images are sorted according to the insertion sequence numbers corresponding to all the images to obtain an image insertion sequence. For an image in the image insertion sequence, the similarity between the image and the image arranged one position after the image is greater than the similarity between the image and the images arranged two positions after the image.

[0124] For example, the image set includes images: r1, r2, r3, ..., r8. After steps S211 to S214, the insertion sequence corresponding to each image in r1, r2, r3, ..., r8 can be obtained, and the image insertion sequence is: r1, r3, r7, r6, r2, r8, r5, r4. Among them, the similarity between r7 and r6 is greater than the similarity between r7 and r2 (r8, r5, r4).

[0125] In the embodiment of the present invention, the greater the similarity between the two images, the more similar the styles between the two images are. By determining the image insertion sequence based on the similarity between any two images in the image set, and playing the images in the order of the image insertion sequence, images with similar styles can be played continuously, making the video picture more harmonious.

[0126] S22: Obtain several beats of the audio.

[0127] In the embodiment of the present invention, the signal corresponding to the audio is analyzed to determine a number of beat points corresponding to the audio. The beat point is the moment when a musical instrument plays a specific note, such as a drum beat. The audio includes a large number of beat points, and any two different beat points correspond to different playing times.

[0128] Specifically, step S22 includes:

[0129] S221. Obtain an initial signal corresponding to the audio.

[0130] In the embodiment of the present invention, the initial signal is a time domain signal corresponding to the audio, the abscissa of the initial signal is time, and the ordinate of the initial signal is energy of the audio signal.

[0131] In an embodiment of the present invention, there is some noise in the initial signal. In order to obtain more accurate beat points in subsequent steps and to reduce the amount of data processed in subsequent steps, the initial signal can be preprocessed first to eliminate interference signals in the initial signal and reduce the amount of data.

[0132] The process of preprocessing the initial signal includes:

[0133] Determine multiple central moments in the initial signal, calculate the cumulative value of all energies in the neighborhood of each central moment, use the cumulative value as the amplitude corresponding to the central moment, and then obtain a preprocessed initial signal, and use the preprocessed initial signal to replace the initial signal. Any two central moments in the multiple central moments may intersect.

[0134] In the embodiment of the present invention, the neighborhood of the central moment may be: from a preset time before the central moment to a neighborhood time after the moment. For example, the central moment is t0, the neighborhood of t0 is [t0-tr, t0+tr], and the neighborhood time may be 10ms.

[0135] Specifically, the initial signal is preprocessed using formula (3).

[0136]

[0137] Among them, in the preprocessed initial signal, Wt is the amplitude corresponding to the central time t; tr is the neighborhood time length, tr can be set to 10ms; An is the amplitude corresponding to time n; t = t + 10ms, that is, the amplitude corresponding to time t is determined every 10ms. For example, the amplitude corresponding to 10ms in the preprocessed initial signal can be determined based on the amplitudes corresponding to all sampling points in [0, 20ms]; since t = t + 10ms, the amplitude corresponding to 20ms in the preprocessed initial signal can be determined based on the amplitudes corresponding to all sampling points in [10ms, 30ms].

[0138] Among them, the amplitude corresponding to the time n can be obtained by the sampling frequency f 0 Determine that the sampling point corresponding to time n is n*f 0 . You can do it in the n*fth 0 The amplitude corresponding to the sampling point is used as the amplitude corresponding to the time n. The sampling frequency can be 50 Hz.

[0139] S222: Perform low-pass filtering on the initial signal to obtain a first signal corresponding to the audio.

[0140] In the embodiment of the present invention, in theory, the beats in music are usually generated by low-frequency instruments, such as percussion instruments (bass drums, hand drums, etc.). Therefore, it is easier to determine several beat points by analyzing the low-frequency signal corresponding to the audio. First, the initial signal is low-pass filtered to filter out signals greater than a low-frequency threshold, and the low-frequency threshold can be: 200Hz. The initial signal can be processed by a Gaussian low-pass filter to obtain the first signal.

[0141] S223. Determine a plurality of target amplitudes according to the first signal.

[0142] In an embodiment of the present invention, the first signal is divided into multiple signal segments, and the maximum amplitude corresponding to each signal segment is obtained. The step of obtaining the maximum amplitude corresponding to each signal segment is performed a preset number of times; for each signal segment, if the maximum amplitude of the signal segment obtained for the preset number of times is the same, the maximum amplitude of the signal segment is used as the target amplitude of the signal segment.

[0143] The specific process of obtaining the maximum amplitude corresponding to each signal segment each time is as follows:

[0144] The signal segment is determined by setting a window of a preset duration. Specifically, multiple signal segments are determined according to the preset duration, and the duration corresponding to each signal segment is the preset duration. For example, if the preset duration corresponding to the window is set to L, then the duration corresponding to each signal segment is L.

[0145] The window slides on the first signal according to a preset step length to determine the signal segment and determine the maximum amplitude in each signal segment. The preset step length is a step length in the time dimension, and the preset step length may be greater than or equal to the preset duration. When the preset step length may be greater than or equal to the preset duration, any two signal segments in the multiple signal segments do not intersect; the preset step length may be less than the preset duration. When the preset step length is less than the preset duration, any two adjacent signal segments in the multiple signal segments intersect.

[0146] In the embodiment of the present invention, the preset value can be set to 20 times. That is to say, for each signal segment, it is necessary to repeat the preset value times to determine the maximum amplitude corresponding to the signal segment. If the preset value maximum amplitudes obtained by the preset value times are the same, the maximum amplitude is used as the target amplitude corresponding to the signal segment; if there are any two different maximum amplitudes among the preset value maximum amplitudes obtained by the preset value times, the signal segment has no target amplitude.

[0147] S224. For each of the target amplitudes, take the time corresponding to the target amplitude as the beat point corresponding to the target amplitude.

[0148] In the embodiment of the present invention, for a target amplitude, the target amplitude is the maximum amplitude in a signal segment of the first signal (the low-frequency signal after filtering out the high frequency); the beat point in the low-frequency signal has the largest loudness (maximum amplitude), so the beat point can be determined according to the target amplitude. The beat point is the moment when a musical instrument plays a specific note, and the moment corresponding to the target amplitude is used as the beat point corresponding to the target amplitude.

[0149] S23: Determine a target beat point corresponding to each image in the image insertion sequence according to the image insertion sequence, the plurality of beat points and the audio.

[0150] In an embodiment of the present invention, the image insertion sequence includes a plurality of images arranged in an insertion order, and a target beat point of each image is determined according to a plurality of beat points, the audio, and the image insertion sequence. For any two adjacent images in the image insertion sequence, the any two adjacent images include a first image played first and a second image played later, when the playback time of the video reaches the target beat point corresponding to the first image, the first image is inserted, and when the playback time of the video reaches the target beat point corresponding to the second image, the second image is inserted.

[0151] Specifically, step S23 includes:

[0152] S231. Obtain the audio duration corresponding to the audio and the number of images; determine the image insertion point corresponding to each image in the image insertion sequence according to the number of images and the audio duration, wherein the duration between two adjacent image insertion points is determined according to the audio duration and the number of images.

[0153] In an embodiment of the present invention, the audio duration is the duration of the audio, and after the video is generated, the duration of the video playback is equal to the audio duration. The number of images refers to the number of images in the image insertion sequence, that is, the number of images in the image set to be processed. The ratio of the audio duration to the number of images can be calculated to obtain the average playback duration of each image. The image insertion point corresponding to each image can be obtained based on the average playback duration and the audio duration. Among them, the image insertion point corresponding to the image ranked first in the image insertion sequence can be set as the start time of audio playback.

[0154] For example, the audio duration is 20 seconds, and the image insertion sequence includes 4 images, then the playing duration of each image is 5 seconds. Assuming that the image insertion sequence includes: g1, g2, g3 and g4, it can be determined that the image insertion point corresponding to g1 is t=0 seconds, the image insertion point corresponding to g2 is t=5 seconds, the image insertion point corresponding to g3 is t=10 seconds, and the image insertion point corresponding to g4 is t=15 seconds.

[0155] S232: Determine target beat points corresponding to each of the plurality of images according to the plurality of image insertion points and the plurality of beat points.

[0156] In an embodiment of the present invention, for each image insertion point, the beat point closest to the image insertion point is determined among the several beat points, and the beat point closest to the image insertion point is used as the target beat point of the image corresponding to the image insertion point.

[0157] In the embodiment of the present invention, the so-called “shortest distance” means that the distance between the time corresponding to the target beat point and the time corresponding to the image insertion point is the shortest.

[0158] For example, several beat points are: {j1, j2, j3, …, j20}, and several image insertion points are: {c1, c2, …, c5}. For cm, if cm = jn, then jn is used as the target beat point of the image corresponding to cm, where 1≦m≦5, 1≦n≦20; if cm≠jn, then jn that is closest to cm is determined, and jn is used as the target beat point of the image corresponding to cm.

[0159] For example, the image insertion point corresponding to g3 is t=10 seconds. If there is a beat point of t=10 seconds, then t=10 seconds is used as the target beat point of the image corresponding to g3. The image insertion point corresponding to g3 is t=10 seconds. Among the beat points, the beat point closest to t=10 seconds is t=11 seconds. Then t=11 seconds is used as the target beat point of the image corresponding to g3.

[0160] S3. Inserting an image corresponding to each target beat point in the audio to generate a video.

[0161] In an embodiment of the present invention, for each target beat point, when the playback time of the audio reaches the target beat point, an image corresponding to the target beat point is inserted, and the image is used as an image frame played between the target beat point and the next target beat point.

[0162] For example, the audio duration is 20 seconds, and the image insertion sequence includes 4 images, then the playing duration of each image is 5 seconds. Assuming that the image insertion sequence includes: g1, g2, g3 and g4, it can be determined that the target beat point corresponding to g1 is t=0 seconds, the target beat point corresponding to g2 is t=5.5 seconds, the target beat point corresponding to g3 is t=9.5 seconds, and the target beat point corresponding to g4 is t=16 seconds. When the audio playing time is 0 seconds, insert g1, and insert g1 in every frame from 0 seconds to 5.5 seconds; when the audio playing time is 5.5 seconds, insert g2, and insert g2 in every frame from 5.5 seconds to 9.5 seconds; when the audio playing time is 9.5 seconds, insert g3, and insert g3 in every frame from 9.5 seconds to 16 seconds; when the audio playing time is 16 seconds, insert g4, and insert g4 in every frame from 16 seconds to 20 seconds.

[0163] In the embodiment of the present invention, the generated video can achieve: when the video is played, the same image is played continuously between two target beat points, and each target beat point is a beat point. For example, in the above example, g1 is played continuously from 0 seconds to 5.5 seconds, switched to g2 at 5.5 seconds, and g2 is played continuously from 5.5 seconds to 9.5 seconds, wherein 5.5 seconds is a beat point, that is, the image played is switched at the beat point.

[0164] Based on the above video generation method, see Figure 3 The embodiment of the present invention further provides a video generating device, comprising:

[0165] An acquisition unit, used to acquire an image set to be processed and an audio to be processed, wherein the image set includes a plurality of images;

[0166] A target beat point determination unit, configured to determine a target beat point corresponding to each of the plurality of images based on the image set and the audio;

[0167] The video generating unit is used to insert an image corresponding to each target beat point in the audio to generate a video.

[0168] In one embodiment, the present invention provides a computer device, which may be a terminal, and the internal structure is as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network model interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network model interface of the computer device is used to communicate with an external terminal through a network model connection. When the computer program is executed by the processor, a method for generating a video is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0169] Those skilled in the art will understand that Figure 4 What is shown is only a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0170] An embodiment of the present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0171] Acquire an image set to be processed and an audio to be processed, wherein the image set includes a plurality of images;

[0172] Based on the image set and the audio, determining a target beat point corresponding to each of the plurality of images;

[0173] An image corresponding to each target beat point in the audio is inserted to generate a video.

[0174] The embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the following steps are implemented:

[0175] Acquire an image set to be processed and an audio to be processed, wherein the image set includes a plurality of images;

[0176] Based on the image set and the audio, determining a target beat point corresponding to each of the plurality of images;

[0177] An image corresponding to each target beat point in the audio is inserted to generate a video.

[0178] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0179] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for generating a video, characterized in that: include: Acquire an image set to be processed and an audio to be processed, wherein the image set includes a plurality of images; Determining a target beat point corresponding to each of the plurality of images based on the image set and the audio, specifically comprising: Based on the similarity between any two images in the image set, the plurality of images are sorted to obtain an image insertion sequence, specifically comprising: Selecting a starting image from the image set, and setting the insertion sequence number of the starting image to the first sequence number; Determine a non-starting image set corresponding to the starting image, wherein the non-starting image set includes a plurality of non-starting images; Based on the similarity between each non-starting image and the starting image, a candidate image corresponding to the starting image is determined, and the insertion sequence number of the candidate image is set to the next sequence number of the insertion sequence number of the starting image; the candidate image is located next to the starting image in the image insertion sequence, and for each non-starting image, the similarity between the non-starting image and the starting image is calculated; the target feature map corresponding to the non-starting image and the target feature map of the starting image are obtained, and the similarity between each non-starting image and the starting image is calculated to obtain the similarity corresponding to each non-starting image, and then a similarity set is obtained; The candidate image is used as a starting image, and the step of determining a non-starting image set corresponding to the starting image is continued until insertion sequence numbers corresponding to all images in the image set are determined; Determining an image insertion sequence corresponding to the image set according to insertion sequence numbers corresponding to all images in the image set; Obtaining a number of beats of the audio; Determining a target beat point corresponding to each image in the image insertion sequence according to the image insertion sequence, the plurality of beat points and the audio, wherein the target beat point is a beat point used for inserting the image among the plurality of beat points of the audio; Inserting an image corresponding to each target beat point in the audio to generate a video; Before obtaining the image set to be processed and determining the image insertion sequence based on the similarity between any two images in the image set, the method further includes: Acquire an original image set, wherein the original image set includes a plurality of original images, and the plurality of original images include at least one template image; Determine a target feature map corresponding to each original image in the original image set; Dividing the original image set into original image subsets of different categories based on all determined target feature maps; Taking any original image subset including the template image as the image set to be processed; Acquire an initial signal corresponding to the audio; Performing low-pass filtering on the initial signal to obtain a first signal corresponding to the audio; determining a plurality of target amplitudes according to the first signal; For each of the target amplitudes, the time corresponding to the target amplitude is used as the beat point corresponding to the target amplitude.

2. The method for generating a video according to claim 1, characterized in that: Determining a non-starting image set corresponding to the starting image specifically includes: For the starting image, all images whose insertion sequence numbers have not been determined are selected from the image set to obtain a non-starting image set corresponding to the starting image.

3. The method for generating a video according to claim 1, characterized in that: Determining a candidate image corresponding to the starting image based on the similarity between each non-starting image in the non-starting image set and the starting image specifically includes: Calculating the similarity between each non-starting image in the non-starting image set and the starting image respectively to obtain a similarity set; A maximum similarity is selected from the similarity set, and an image corresponding to the maximum similarity is used as a candidate image corresponding to the starting image.

4. The method for generating a video according to claim 1, characterized in that: The step of determining the target beat point corresponding to each image in the image insertion sequence according to the image insertion sequence, the plurality of beat points and the audio specifically includes: Obtaining the audio duration corresponding to the audio and the number of images of the plurality of images; Determining an image insertion point corresponding to each image in the image insertion sequence according to the number of images and the audio duration, wherein the duration between two adjacent image insertion points is determined according to the audio duration and the number of images; The target beat points corresponding to each of the plurality of images are determined according to the plurality of image insertion points and the plurality of beat points.

5. The method for generating a video according to claim 4, characterized in that: The step of determining the target beat points corresponding to the plurality of images according to the plurality of image insertion points and the plurality of beat points specifically includes: For each image insertion point, a beat point closest to the image insertion point is determined among the plurality of beat points, and the beat point closest to the image insertion point is used as a target beat point of the image corresponding to the image insertion point.

6. The method for generating a video according to claim 1, characterized in that: Inserting an image corresponding to each target beat point in the audio to generate a video specifically includes: For each target beat point, when the playing time of the audio reaches the target beat point, an image corresponding to the target beat point is inserted, and the image is used as an image frame played between the target beat point and the next target beat point.

7. A video generating device, characterized in that: include: An acquisition unit is used to acquire an image set to be processed and an audio to be processed, wherein the image set includes a plurality of images; acquire an original image set, wherein the original image set includes a plurality of original images, and the plurality of original images include at least one template image; determine a target feature map corresponding to each original image in the original image set; divide the original image set into original image subsets of different categories based on all determined target feature maps; use any original image subset including the template image as the image set to be processed; sort the plurality of images based on the similarity between any two images in the image set to obtain an image insertion sequence; select a starting image from the image set, and set the insertion sequence number of the starting image to the first sequence number; determine a non-starting image set corresponding to the starting image, wherein the non-starting image set includes a plurality of non-starting images; based on each non-starting image, The similarity between the image and the starting image determines the candidate image corresponding to the starting image, and the insertion sequence number of the candidate image is set to the next sequence number of the insertion sequence number of the starting image; the candidate image is located next to the starting image in the image insertion sequence, and for each non-starting image, the similarity between the non-starting image and the starting image is calculated; the target feature map corresponding to the non-starting image and the target feature map of the starting image are obtained, and the similarity between each non-starting image and the starting image is calculated to obtain the similarity corresponding to each non-starting image, and then a similarity set is obtained; the candidate image is used as the starting image, and the step of determining the non-starting image set corresponding to the starting image is continued until the insertion sequence numbers corresponding to all the images in the image set are determined; the image insertion sequence corresponding to the image set is determined according to the insertion sequence numbers corresponding to all the images in the image set; A target beat point determination unit is used to determine the target beat point corresponding to each of the plurality of images based on the image set and the audio; obtain an initial signal corresponding to the audio; perform low-pass filtering on the initial signal to obtain a first signal corresponding to the audio; determine a plurality of target amplitudes according to the first signal; for each of the plurality of target amplitudes, use the moment corresponding to the target amplitude as the beat point corresponding to the target amplitude; obtain a plurality of beat points of the audio; determine the target beat point corresponding to each of the images in the image insertion sequence according to the image insertion sequence, the plurality of beat points and the audio, wherein the target beat point is a beat point used for inserting an image among the plurality of beat points of the audio; The video generating unit is used to insert an image corresponding to each target beat point in the audio to generate a video.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps in the method for generating a video according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps in the video generation method according to any one of claims 1 to 6 are implemented.

Images