Audio-video matching method and storage medium

Abstract

The application discloses an audio-video matching method and a storage medium. The method comprises the following steps: creating tags, and associating corresponding tags with each song audio and each background video respectively; calculating the tag weight of each tag according to the total number of song audios and the number of song audios associated with each tag; when playing a to-be-matched song audio, matching the corresponding candidate background video according to the tag of the to-be-matched song audio, and determining the video weight of each candidate background video according to the tag weight; and selecting a candidate background video as the background video matched with the to-be-matched song audio according to the video weight of each candidate background video. The application can improve the matching accuracy, and make the matched background video more consistent with the song itself.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to an audio-video matching method and a storage medium. BACKGROUND

[0002] At present, when singing a song in a digital audio-visual place, some songs only have audio, in order to make the user have a better visual experience in the singing process, it is necessary to match a background video for the user; or the picture resolution of some old songs is too low to match the existing large-screen display device, and the displayed picture is too blurred, so it is necessary to match a background video for the old songs to achieve good audio-visual effect.

[0003] The traditional method is mainly to randomly match a background video for a song audio, and the background video is often a landscape picture, etc., so that the background video has nothing to do with the song audio, and the user experience is poor. Recently, it has also been found that the song audio and the background video are matched through the tags, if both have the same tag, then the matching is successful directly. Although this method is improved compared with the traditional method, the matching effect is still relatively rough. Moreover, a song often has multiple tags, if each tag is matched with a background video according to equal probability, then it is obviously unreasonable. How to match a suitable background video for the current audio so that the background video situation can be consistent with the song emotion and style as much as possible becomes a problem to be solved. SUMMARY

[0004] The technical problem to be solved by the present application is to provide an audio-video matching method and a storage medium, which can improve the matching accuracy.

[0005] In order to solve the above technical problem, the technical scheme adopted by the present application is as follows: an audio-video matching method, comprising:

[0006] creating tags, and respectively associating corresponding tags for each song audio and each background video;

[0007] According to the total number of song audios and the number of song audios associated with each tag, the tag weight of each tag is calculated respectively;

[0008] When playing a to-be-matched song audio, according to the tag of the to-be-matched song audio, a corresponding candidate background video is matched, and according to the tag weight, the video weight of each candidate background video is determined;

[0009] According to the video weight of each candidate background video, a candidate background video is selected as the background video matched with the to-be-matched song audio.

[0010] The present application also proposes a computer-readable storage medium, which stores a computer program, and the program is executed by a processor to realize the method as described above.

[0011] The application has the beneficial effect that: by calculating the label weight on the basis of label matching, determining the video weight, and selecting the matched video according to the video weight, the matching result is more accurate, when a song audio has multiple labels, which label is more important can be calculated, so that the matched background video is more in line with the song itself situation, thereby achieving the best audio-visual effect and improving the user experience. BRIEF DESCRIPTION OF DRAWINGS

[0012] Figure 1 A flow chart of the audio-video matching method of the application;

[0013] Figure 2 A flow chart of the method of the first embodiment of the application;

[0014] Figure 3 A label weight schematic diagram of the first embodiment of the application. DETAILED DESCRIPTION

[0015] To make the technical content, the achieved purposes and effects of the application clear, the following will be described in detail in combination with the embodiments and the drawings.

[0016] Please refer to Figure 1 An audio-video matching method, comprising:

[0017] Creating labels, and respectively associating corresponding labels with each song audio and each background video;

[0018] According to the total number of song audios and the number of song audios associated with each label, respectively calculating the label weight of each label;

[0019] When playing a to-be-matched song audio, according to the label of the to-be-matched song audio, a corresponding candidate background video is matched, and according to the label weight, the video weight of each candidate background video is determined;

[0020] According to the video weight of each candidate background video, a candidate background video is selected as the background video matched with the to-be-matched song audio.

[0021] From the above description, the application has the beneficial effect that: the matching accuracy can be improved, so that the matched background video is more in line with the song itself situation.

[0022] Further, according to the video weight of each candidate background video, a candidate background video is selected as the background video matched with the to-be-matched song audio, specifically:

[0023] According to the video weight of each candidate background video, the hit probability of each candidate background video is calculated;

[0024] According to the hit probability of each candidate background video, a candidate background video is selected as the background video matched with the to-be-matched song audio.

[0025] From the above description, on the basis of calculating the video weight, the hit probability of the candidate background video is further calculated, and the background video suitable for the song audio is selected according to the hit probability of the candidate background video, so that the matching of the song audio and the background video is more accurate.

[0026] Further, the hit probability of each candidate background video is calculated according to the video weight of each candidate background video, specifically:

[0027] According to the hit probability calculation formula, the hit probability of each candidate background video is calculated, and the hit probability calculation formula is

[0028]

[0029] Wherein, P i represents the hit probability of the i th candidate background video, W i represents the video weight of the i th candidate background video, and M represents the total number of candidate background videos.

[0030] Further, the label weight of each label is calculated according to the total number of song audios and the number of song audios associated with each label, specifically:

[0031] According to the label weight calculation formula, the label weight of each label is calculated, and the label weight calculation formula is w i = lg (N / n i ), wherein w i represents the label weight of the i th label, N represents the total number of song audios, n i represents the number of song audios associated with the i th label.

[0032] From the above description, based on the principle of inverse document frequency, the label weight of each label is calculated. The fewer the number of song audios associated with a label, the higher the rarity of the label, the more personalized it is, and the easier it is to be identified, so the label weight of the label is higher; on the contrary, the more the number of song audios associated with a label, the lower the label weight of the label.

[0033] Further, when playing the to-be-matched song audio, the corresponding candidate background video is matched according to the label of the to-be-matched song audio, and the video weight of each candidate background video is determined according to the label weight, specifically:

[0034] When the audio of the to-be-matched song is played, according to the tags of the audio of the to-be-matched song, a background video associated with the same tag as the audio of the to-be-matched song is obtained as a candidate background video corresponding to the audio of the to-be-matched song, and according to the tag weight of the same tag of each candidate background video and the audio of the to-be-matched song, a video weight of each candidate background video is determined.

[0035] As can be known from the above description, if a tag of the background audio exists in a tag of the audio of the to-be-matched song, the background audio is taken as a candidate background video, and the video weight of the background audio is determined according to the tag weight of the tag of the audio of the to-be-matched song.

[0036] Further, after the video weight of each candidate background video is determined according to the tag weight, the method further comprises:

[0037] According to interval data between the latest playing time of each candidate background video on the current playing terminal and the current time, the video weight of each candidate background video is adjusted, and the interval data comprises interval time length and / or interval song quantity.

[0038] Further, the video weight of each candidate background video is adjusted according to the interval data between the latest playing time of each candidate background video on the current playing terminal and the current time, and specifically:

[0039] If the interval data between the latest playing time of a candidate background video on the current playing terminal and the current time is less than or equal to a preset interval threshold, a weight reduction factor corresponding to the candidate background video is calculated according to the interval data and the interval threshold, and the video weight of the candidate background video is reduced according to the weight reduction factor.

[0040] As can be known from the above description, by reducing the video weight of a candidate background video that has been played for a short time, it is avoided that the user repeatedly sees the same background video for a short time, and the user experience is further improved.

[0041] Further, the tags comprise tags of multiple dimensions, and the multiple dimensions comprise any two or more of the following: emotional dimension, age dimension, language dimension, festival dimension and style dimension.

[0042] As can be known from the above description, by creating tags of multiple dimensions, the song is interpreted in multiple dimensions, and the situation of the song is described as much as possible, so that a background video that matches the situation of the song can be matched subsequently.

[0043] Further, after the candidate background video is selected as the background video matched with the audio of the to-be-matched song according to the video weight of each candidate background video, the method further comprises:

[0044] According to the to-be-matched song audio and the matched background video thereof, a song audio-video file is generated.

[0045] As can be seen from the above description, by synthesizing the matched song audio and the background video, a song audio-video file is obtained, and when the user next requests the corresponding song audio, the already-made song audio-video file can be directly called.

[0046] The application further provides a computer readable storage medium having a computer program stored thereon, and the program is executed by a processor to implement the method described above.

[0047] Embodiment one

[0048] Please refer to Figures 2-3 Embodiment one of the application is an audio-video matching method, which can be applied to audio-visual places such as KTV.

[0049] As Figure 2 shown, the method comprises the following steps:

[0050] S1: creating a label and respectively associating the label with each song audio and each background video, i.e., labeling all song audios and background videos, and each song audio and background video can be associated with more than one label.

[0051] In this embodiment, a set of labels suitable for song classification is established, and the songs are interpreted from multiple dimensions, including emotional dimension, age dimension, language dimension, holiday dimension and style dimension, and each dimension is further subdivided, for example, the labels of the emotional dimension can include “sadness”, “happiness”, “passion” and the like; the labels of the age dimension can include 1960s, 1970s, 1980s, 1990s and the like; and the labels of the style dimension can include blessing songs, children's songs, popular songs and the like.

[0052] S2: calculating the label weight of each label according to the total number of song audios and the number of song audios associated with each label.

[0053] Specifically, the label weight of each label is calculated according to the label weight calculation formula w i = lg(N / n i i), wherein w i represents the label weight of the i-th label, i=1, 2, …, m, m is the total number of labels, N represents the total number of song audios, and n i i represents the number of song audios associated with the i-th label.

[0054] The basic principle of the calculation formula is based on the formula of Inverse Document Frequency (IDF), i.e.,

[0055]

[0056] where |D| represents the total number of files in the corpus, |{j: t i ∈d j} represents the number of files containing the word t i . If a word is more common, the more files containing the word, the larger the denominator in the formula, the smaller the inverse document frequency.

[0057] In this embodiment, the fewer the number of song audios associated with a tag, the higher the rarity of the tag, the more personalized, and the easier to be identified, which is a summary of objective laws, and the background video with personalized tag is also more easily identified by human emotion. Therefore, in this embodiment, the fewer the number of song audios associated with a tag, the higher the tag weight of the tag; on the contrary, the more the number of song audios associated with a tag, the lower the tag weight of the tag.

[0058] S3: When playing the to-be-matched song audio, corresponding candidate background videos are matched according to the tags of the to-be-matched song audio, and video weights of the candidate background videos are determined according to the tag weights.

[0059] Specifically, when playing the to-be-matched song audio, the background videos associated with the same tag as the to-be-matched song audio are obtained as the corresponding candidate background videos of the to-be-matched song audio according to the tags of the to-be-matched song audio, and the video weights of the candidate background videos are determined according to the tag weights of the same tag associated with the to-be-matched song audio and the candidate background videos, respectively.

[0060] For example, when a terminal plays a song, if the song has only audio but no video, a group of background videos are matched according to the tags of the song audio. Assuming that the tags of the song include tag 1 and tag 2, the background videos containing tag 1 or tag 2 are searched as the matching results, the video weight of the background video containing tag 1 is the tag weight of tag 1, and the video weight of the background video containing tag 2 is the tag weight of tag 2.

[0061] If a background video (such as background video a) also contains tag 1 and tag 2, the matching results of the song contain two background videos a, one of which has a video weight of the tag weight of tag 1, and the other of which has a video weight of the tag weight of tag 2.

[0062] S4: Calculate the hit probability of each candidate background video according to the video weight of each candidate background video.

[0063] Specifically, the hit probability of each candidate background video is calculated according to the hit probability calculation formula, and the hit probability calculation formula is:

[0064]

[0065] wherein P i represents the hit probability of the i-th candidate background video, i = 1, 2, …, M, M represents the total number of candidate background videos, W i represents the video weight of the i-th candidate background video.

[0066] That is, the video weight of a certain candidate background video is divided by the sum of the video weights of all candidate background videos, and the hit probability of the candidate background video is obtained.

[0067] S5: According to the hit probability of each candidate background video, a candidate background video is selected as the background video matched with the to-be-matched song audio.

[0068] That is, according to the hit probability calculated in step S4, the background video is matched with the to-be-matched song audio, and the higher the hit probability of the background video, the higher the probability of being matched.

[0069] For example, in an actual application scenario, the created tags can include inspirational, blessing songs, dance songs, children's songs, happy home, opera, popular songs, etc. According to these tags, all songs in the song library are tagged, and all background videos are tagged.

[0070] Suppose the total number of songs is 50,000, of which the number of songs with the "children's songs" tag is 366, then the tag weight of "children's songs" is lg(50000 / 366) = 2.13; the number of songs with the "popular songs" tag is 41,000, then the tag weight of "popular songs" is lg(50000 / 41000) = 0.08. Similarly, the tag weights of various tags are calculated as shown in Table 1. Figure 3

[0071] Suppose "catching a catfish" is played, and the song has both "children's songs" and "popular songs" tags. At this time, background videos containing "children's songs" or "popular songs" tags are obtained, and suppose there is only one background video containing the "children's songs" tag and only one background video containing the "popular songs" tag. Then, the two background videos are obtained, which are respectively called children's songs video and popular songs video.

[0072] According to Figure 3 ​It can be seen that the label weight of "children's songs" is 2.13 and the label weight of "popular songs" is 0.08, so the video weight of the children's songs video is 2.13 and the video weight of the popular songs video is 0.08, the hit probability of the children's songs video is 2.13 / (2.13+0.08)=0.96, and the hit probability of the popular songs video is 0.08 / (2.13+0.08)=0.04.

[0073] The playing terminal matches the video according to the above hit probability. The hit probability of the children's songs video is larger, so the probability of being matched is much larger than that of the popular songs video.

[0074] Further, a song audio-video file is generated according to the to-be-matched song audio and the matched background video, that is, the to-be-matched song audio and the matched background video are synthesized, and when the user next on-demand plays the to-be-matched song audio, the corresponding song audio-video file can be directly called.

[0075] The embodiment establishes a label system suitable for KTV songs, interprets the songs in multiple dimensions, labels the song audio and the background video respectively, calculates the label weight of each label based on the principle of inverse document frequency, reads all labels of the song audio when matching the background video for the song audio, acquires the background video corresponding to the labels, calculates the hit probability of each background video, and finally matches according to the hit probability, so as to realize the matching of the song audio and the background video.

[0076] The embodiment fuses the method of inverse document frequency IDF on the basis of the traditional label matching, calculates the weight of the label, calculates the matching probability based on the label weight, and makes the matching result more accurate. When a song has multiple labels, it can be calculated which label is more important, so that the matched background video is more in line with the song itself situation.

[0077] Embodiment Two

[0078] The embodiment is a further expansion of the embodiment one, and the same parts will not be repeated. The difference is that in the embodiment, after the video weight of each candidate background video is calculated, the video weight of each candidate background video is adjusted according to the interval data between the latest playing time of each candidate background video on the current playing terminal and the current time, wherein the interval data can be interval time length or interval song quantity.

[0079] Specifically, if the interval data between the latest playing time of a candidate background video on the current playing terminal and the current time is less than or equal to a preset interval threshold, a weight reduction factor corresponding to the candidate background video is calculated according to the interval data and the interval threshold, and the video weight of the candidate background video is reduced according to the weight reduction factor.

[0080] For example, if the interval between the time when a certain candidate background video was played on the current playing terminal last time and the current time is no more than 16-20 minutes, or the number of other songs played between them is no more than 4-5, the video weight of the candidate background video is reduced.

[0081] In this embodiment, the quotient obtained by dividing the interval data by the interval threshold value is taken as the weight reduction factor, so the value range of the weight reduction factor is (0, 1]. Then the original video weight of the candidate background video is multiplied by the weight reduction factor to obtain the video weight of the candidate background video after weight reduction.

[0082] In this embodiment, by reducing the weight of the candidate background video played in a short time, the user can avoid seeing the same background video repeatedly in a short time, and the user experience is further improved.

[0083] Embodiment Three

[0084] This embodiment is a computer readable storage medium corresponding to the above-mentioned embodiments, which stores a computer program, the program is executed by a processor to realize each step of the audio and video matching method in the above-mentioned embodiments, and the same technical effect can be achieved, which will not be repeated here.

[0085] In summary, the audio and video matching method and storage medium provided by the present application calculate the label weight on the basis of label matching, determine the video weight, and calculate the hit probability based on the weight, so that the matching result is more accurate. When a song audio has multiple labels, it can be calculated which label is more important, so that the matching background video is more in line with the song itself situation.

[0086] The above-mentioned is only an embodiment of the present application, and does not limit the patent range of the present application, and any equivalent transformation or direct or indirect application in related technical fields based on the content of the specification and drawings of the present application is also included in the patent protection range of the present application.

Claims

1. An audio-video matching method, characterized by, The method comprises the following steps: creating tags, and associating corresponding tags with each song audio and each background video respectively; calculating the tag weight of each tag according to the total number of song audios and the number of song audios associated with each tag; when playing a to-be-matched song audio, matching the corresponding candidate background video according to the tag of the to-be-matched song audio, and determining the video weight of each candidate background video according to the tag weight; selecting a candidate background video as the background video matched with the to-be-matched song audio according to the video weight of each candidate background video. The method of selecting a candidate background video as the background video matched with the to-be-matched song audio according to the video weight of each candidate background video specifically comprises: calculating the hit probability of each candidate background video according to the video weight of each candidate background video; selecting a candidate background video as the background video matched with the to-be-matched song audio according to the hit probability of each candidate background video.

2. The audio-video matching method of claim 1, wherein, The method of calculating the hit probability of each candidate background video according to the video weight of each candidate background video specifically comprises: calculating the hit probability of each candidate background video according to a hit probability calculation formula, wherein the hit probability calculation formula is where P i represents the hit probability of the i-th candidate background video, W i represents the video weight of the i-th candidate background video, and M represents the total number of candidate background videos.

3. The audio-video matching method of claim 1, wherein, The method of calculating the tag weight of each tag according to the total number of song audios and the number of song audios associated with each tag specifically comprises: According to a label weight calculation formula, label weights of the labels are respectively calculated, the label weight calculation formula being w i =lg(N / n i ), wherein w i represents the label weight of the i-th label, N represents the total number of song audios, and n i represents the number of song audios associated with the i-th label.

4. The audio-video matching method of claim 1, wherein, The method of matching the corresponding candidate background video according to the tag of the to-be-matched song audio and determining the video weight of each candidate background video according to the tag weight when playing the to-be-matched song audio specifically comprises: when playing the to-be-matched song audio, obtaining the background video associated with the same tag as the to-be-matched song audio as the candidate background video corresponding to the to-be-matched song audio according to each tag of the to-be-matched song audio, and determining the video weight of each candidate background video according to the tag weight of the same tag associated with the to-be-matched song audio and each candidate background video respectively.

5. The audio-video matching method of claim 1, wherein, After determining the video weight of each candidate background video according to the tag weight, the method further comprises: adjusting the video weight of each candidate background video according to the interval data between the latest playing time of each candidate background video on the current playing terminal and the current time, wherein the interval data comprises interval time length and / or interval song quantity. The method of adjusting the video weight of each candidate background video according to the interval data between the latest playing time of each candidate background video on the current playing terminal and the current time specifically comprises: if the interval data between the latest playing time of a candidate background video on the current playing terminal and the current time is less than or equal to a preset interval threshold, calculating a weight reduction factor corresponding to the candidate background video according to the interval data and the interval threshold, and reducing the video weight of the candidate background video according to the weight reduction factor.

7. The audio-video matching method of any of claims 1-6, wherein, The tags comprise tags of multiple dimensions, and the multiple dimensions comprise any two or more of the following: emotional dimension, age dimension, language dimension, festival dimension, and style dimension.

8. The audio-video matching method according to any of claims 1-6, wherein, After the candidate background video is selected as the background video matched with the to-be-matched song audio according to the video weight of each candidate background video, the method further comprises: Generating a song audio-video file according to the to-be-matched song audio and the matched background video.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the method of any one of claims 1-8.

Images