Music video generation method, device, electronic device and storage medium

Abstract

The present disclosure relates to a music video generation method, device, electronic device and storage medium, the method comprising: obtaining a lyrics sequence, a chord information set and a rhythm information set of a target music, determining a lyrics semantic information sequence based on the lyrics sequence, the number of lyrics semantic information in the lyrics semantic information sequence being the same as the number of lyrics in the lyrics sequence, generating an initial picture set based on the lyrics semantic information sequence and the rhythm information set, performing style conversion on pictures in the initial picture set using the chord information set to obtain a target picture set, determining a target video based on the target picture set and the target music, the target video including the target music. The present application obtains more pictures that fit the text and rhythm through the combination of music rhythm, chords and text, and on this basis, further performs style conversion on pictures using chord information to obtain video pictures with richer information and that fit the music better.

Description

Technical Field

[0001] The present disclosure relates to the field of Internet technology, and in particular to a music video generation method, device, electronic device and storage medium. Background Art

[0002] With the rapid development of short video platforms and the rapid improvement of mobile phone hardware performance, video creation is becoming more and more popular and popular among the audience. Traditional text content and music content are combined with media that can carry more information. In the existing technology, there are many technologies related to video soundtracks in the field of short videos, but the technology for adapting music to the picture is indeed rare. Basically, a single mapping of video material library and music emotion, or material library and style is adopted, which cannot carry more information to make the music and picture more suitable. Summary of the invention

[0003] The present disclosure provides a music video generation method, device, electronic device and storage medium. The technical solution of the present disclosure is as follows:

[0004] According to a first aspect of an embodiment of the present disclosure, there is provided a method for generating a music video, comprising:

[0005] Obtaining a lyrics sequence, a chord information set, and a rhythm information set of the target music;

[0006] Determining a lyrics semantic information sequence based on the lyrics sequence; the amount of lyrics semantic information in the lyrics semantic information sequence is the same as the number of lyrics in the lyrics sequence;

[0007] Generate an initial image set based on the lyrics semantic information sequence and rhythm information set;

[0008] Use the chord information set to perform style conversion on the pictures in the initial picture set to obtain the target picture set;

[0009] A target video is determined based on the target picture set and the target music; the target video includes the target music.

[0010] In some possible embodiments, obtaining a chord information set of a target music includes:

[0011] Analyze the chords of the target music to obtain a plurality of chord progression information arranged in the order of playing the target music, and a time period of a music clip corresponding to each chord progression information in the target music;

[0012] Determine a plurality of style information corresponding to a plurality of chord progression information one by one;

[0013] The chord information set of the target music is determined based on the style information corresponding to each chord progression information and the time period of the music segment corresponding to each chord progression information in the target music.

[0014] In some possible embodiments, obtaining a rhythm information set of target music includes:

[0015] Perform spectrum conversion on the target music to obtain the Mel spectrum corresponding to the target music;

[0016] The mel spectrum corresponding to the target music is subjected to impact separation processing to obtain a rhythm spectrum and a chord spectrum; the rhythm spectrum represents the regular changes in the rhythm of the target music, and the chord spectrum represents the chord melody of the target music;

[0017] Extract the amplitude variation of the rhythm spectrum to obtain the amplitude variation data of the target music;

[0018] Extract momentum changes from the chord spectrum to obtain momentum change data of the target music;

[0019] A rhythm information set is determined based on the amplitude change data and momentum change data of the target music; the rhythm information set includes multiple rhythm information and a timestamp of each rhythm information in the target music; each rhythm information represents a frame energy value of the target music.

[0020] In some possible embodiments, generating an initial picture set according to a lyrics semantic information sequence and a rhythm information set includes:

[0021] For each lyric semantic information in the lyric semantic information sequence, the following is performed: determining a time period of the lyrics corresponding to the lyric semantic information being processed in the target music;

[0022] Based on the time period of the lyrics corresponding to the lyrics semantic information in the target music, determining the target rhythm information associated with the lyrics corresponding to the lyrics semantic information and the timestamp corresponding to the target rhythm information from the rhythm information set;

[0023] Determine a first picture set corresponding to the lyrics semantic information based on the lyrics semantic information, the target rhythm information, the timestamp corresponding to the target rhythm information, the time period of the lyrics corresponding to the lyrics semantic information in the target music, and the preset picture frame rate;

[0024] When the first picture set corresponding to each lyric semantic information is determined, the first picture set corresponding to each lyric semantic information is arranged in a preset order to obtain an initial picture set; the preset order is the time sequence of the lyrics corresponding to each lyric semantic information in the target music.

[0025] In some possible embodiments, determining the first picture set corresponding to the word semantic information based on the lyrics semantic information, the target rhythm information, the timestamp corresponding to the target rhythm information, the time period of the lyrics corresponding to the lyrics semantic information in the target music, and the preset picture frame rate includes:

[0026] Determine the number of pictures in the first picture set corresponding to the lyrics semantic information based on the time period of the lyrics corresponding to the lyrics semantic information in the target music and the preset picture frame rate;

[0027] Determine a semantic information index corresponding to each picture in the first picture set according to the target rhythm information, a timestamp corresponding to the target rhythm information, and the number of pictures in the first picture set; the semantic information index represents a picture screen change speed of each picture in the first picture set;

[0028] The first picture set is generated based on the semantic information index corresponding to each picture in the first picture set and the lyrics semantic information.

[0029] In some possible embodiments, determining a semantic information index corresponding to each picture in the first picture set according to the target rhythm information, a timestamp corresponding to the target rhythm information, and the number of pictures in the first picture set includes:

[0030] Determine a first semantic information index and a second semantic information index corresponding to each picture in the first picture set according to the target rhythm information, the timestamp corresponding to the target rhythm information, and the number of pictures in the first picture set; the first semantic information index corresponds to the lyrics semantic information; the second semantic information index corresponds to the adjacent lyrics voice information of the lyrics semantic information; the first semantic information index and the second semantic information index comprehensively represent the picture screen change speed of each picture in the first picture set;

[0031] Generating the first picture set based on the semantic information index corresponding to each picture in the first picture set and the lyrics semantic information includes:

[0032] Determine the first semantic information corresponding to each picture based on the first semantic information index corresponding to each picture and the lyrics semantic information;

[0033] Determine the second semantic information corresponding to each picture based on the second semantic information index corresponding to each picture and the semantic information of adjacent lyrics;

[0034] Determine comprehensive semantic information corresponding to each picture based on the first semantic information and the second semantic information corresponding to each picture;

[0035] Generate a first picture set based on comprehensive semantic information corresponding to each picture;

[0036] The adjacent lyrics semantic information is the previous lyrics semantic information or the next lyrics semantic information of the lyrics semantic information in the lyrics semantic information sequence.

[0037] In some possible embodiments, style conversion is performed on pictures in the initial picture set using the chord information set to obtain a target picture set, including:

[0038] Based on the time period of each lyric in the target music, the style information corresponding to each chord progression information in the chord information set, and the time period of each chord progression information in the target music, determine the target style information corresponding to each first picture set in the initial picture set and the time period corresponding to the target style information; the first picture set is the picture set corresponding to each lyric semantic information in the initial picture set;

[0039] Performing style conversion on each picture in each first picture set based on the target style information and the time period corresponding to the target style information, to obtain a second picture set corresponding to each lyric semantic information; the pictures in the second picture set correspond to the pictures in the initial picture set one-to-one, and the two pictures that correspond one-to-one have the same picture content but different picture styles;

[0040] Arrange the second picture set in the order of playing the target music to obtain the target picture set.

[0041] According to a second aspect of an embodiment of the present disclosure, there is provided a music video generating device, comprising:

[0042] An information acquisition module is configured to acquire a lyrics sequence, a chord information set, and a rhythm information set of a target music;

[0043] The information sequence acquisition module is configured to determine the lyrics semantic information sequence based on the lyrics sequence; the number of lyrics semantic information in the lyrics semantic information sequence is the same as the number of lyrics in the lyrics sequence;

[0044] A picture set generation module is configured to generate an initial picture set according to the lyrics semantic information sequence and the rhythm information set;

[0045] A style conversion module is configured to perform style conversion on the pictures in the initial picture set using the chord information set to obtain a target picture set;

[0046] The video determination module is configured to determine a target video based on a target picture set and target music; the target video includes target music.

[0047] In some possible embodiments, the information acquisition module is configured to execute:

[0048] Analyze the chords of the target music to obtain a plurality of chord progression information arranged in the order of playing the target music, and a time period of a music clip corresponding to each chord progression information in the target music;

[0049] Determine a plurality of style information corresponding to a plurality of chord progression information one by one;

[0050] The chord information set of the target music is determined based on the style information corresponding to each chord progression information and the time period of the music segment corresponding to each chord progression information in the target music.

[0051] In some possible embodiments, the information acquisition module is configured to execute:

[0052] Perform spectrum conversion on the target music to obtain the Mel spectrum corresponding to the target music;

[0053] The mel spectrum corresponding to the target music is subjected to impact separation processing to obtain a rhythm spectrum and a chord spectrum; the rhythm spectrum represents the regular changes in the rhythm of the target music, and the chord spectrum represents the chord melody of the target music;

[0054] Extract the amplitude variation of the rhythm spectrum to obtain the amplitude variation data of the target music;

[0055] Extract momentum changes from the chord spectrum to obtain momentum change data of the target music;

[0056] A rhythm information set is determined based on the amplitude change data and momentum change data of the target music; the rhythm information set includes multiple rhythm information and a timestamp of each rhythm information in the target music; each rhythm information represents a frame energy value of the target music.

[0057] In some possible embodiments, the picture set generation module is configured to execute:

[0058] For each lyric semantic information in the lyric semantic information sequence, the following is performed: determining a time period of the lyrics corresponding to the lyric semantic information being processed in the target music;

[0059] Based on the time period of the lyrics corresponding to the lyrics semantic information in the target music, determining the target rhythm information associated with the lyrics corresponding to the lyrics semantic information and the timestamp corresponding to the target rhythm information from the rhythm information set;

[0060] Determine a first picture set corresponding to the lyrics semantic information based on the lyrics semantic information, the target rhythm information, the timestamp corresponding to the target rhythm information, the time period of the lyrics corresponding to the lyrics semantic information in the target music, and the preset picture frame rate;

[0061] When the first picture set corresponding to each lyric semantic information is determined, the first picture set corresponding to each lyric semantic information is arranged in a preset order to obtain an initial picture set; the preset order is the time sequence of the lyrics corresponding to each lyric semantic information in the target music.

[0062] In some possible embodiments, the picture set generation module is configured to execute:

[0063] Determine the number of pictures in the first picture set corresponding to the lyrics semantic information based on the time period of the lyrics corresponding to the lyrics semantic information in the target music and the preset picture frame rate;

[0064] Determine a semantic information index corresponding to each picture in the first picture set according to the target rhythm information, a timestamp corresponding to the target rhythm information, and the number of pictures in the first picture set; the semantic information index represents a picture screen change speed of each picture in the first picture set;

[0065] The first picture set is generated based on the semantic information index corresponding to each picture in the first picture set and the lyrics semantic information.

[0066] In some possible embodiments, the picture set generation module is configured to execute:

[0067] Determine a first semantic information index and a second semantic information index corresponding to each picture in the first picture set according to the target rhythm information, the timestamp corresponding to the target rhythm information, and the number of pictures in the first picture set; the first semantic information index corresponds to the lyrics semantic information; the second semantic information index corresponds to the adjacent lyrics voice information of the lyrics semantic information; the first semantic information index and the second semantic information index comprehensively represent the picture screen change speed of each picture in the first picture set;

[0068] Generating the first picture set based on the semantic information index corresponding to each picture in the first picture set and the lyrics semantic information includes:

[0069] Determine the first semantic information corresponding to each picture based on the first semantic information index corresponding to each picture and the lyrics semantic information;

[0070] Determine the second semantic information corresponding to each picture based on the second semantic information index corresponding to each picture and the semantic information of adjacent lyrics;

[0071] Determine comprehensive semantic information corresponding to each picture based on the first semantic information and the second semantic information corresponding to each picture;

[0072] Generate a first picture set based on comprehensive semantic information corresponding to each picture;

[0073] The adjacent lyrics semantic information is the previous lyrics semantic information or the next lyrics semantic information of the lyrics semantic information in the lyrics semantic information sequence.

[0074] In some possible embodiments, the style transfer module is configured to perform:

[0075] Based on the time period of each lyric in the target music, the style information corresponding to each chord progression information in the chord information set, and the time period of each chord progression information in the target music, determine the target style information corresponding to each first picture set in the initial picture set and the time period corresponding to the target style information; the first picture set is the picture set corresponding to each lyric semantic information in the initial picture set;

[0076] Performing style conversion on each picture in each first picture set based on the target style information and the time period corresponding to the target style information, to obtain a second picture set corresponding to each lyric semantic information; the pictures in the second picture set correspond to the pictures in the initial picture set one-to-one, and the two pictures that correspond one-to-one have the same picture content but different picture styles;

[0077] Arrange the second picture set in the order of playing the target music to obtain the target picture set.

[0078] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute instructions to implement a method as described in any one of the first or second aspects above.

[0079] According to the fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute any method in the first aspect or the second aspect of the embodiment of the present disclosure.

[0080] According to the fifth aspect of an embodiment of the present disclosure, a computer program product is provided, the computer program product comprising a computer program, the computer program being stored in a readable storage medium, at least one processor of a computer device reading and executing the computer program from the readable storage medium, so that the computer device performs any one of the methods of the first aspect or the second aspect of the embodiment of the present disclosure.

[0081] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0082] Obtain the lyrics sequence, chord information set and rhythm information set of the target music, determine the lyrics semantic information sequence based on the lyrics sequence, the number of lyrics semantic information in the lyrics semantic information sequence is the same as the number of lyrics in the lyrics sequence, generate an initial picture set based on the lyrics semantic information sequence and the rhythm information set, use the chord information set to perform style conversion on the pictures in the initial picture set to obtain the target picture set, determine the target video based on the target picture set and the target music, and the target video includes the target music. This application obtains more pictures that fit the text and rhythm through the combination of music rhythm, chords and text, and on this basis, further style conversion is performed on the pictures using chord information to obtain video pictures with richer information and more fit to the music.

[0083] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0085] Figure 1 is a schematic diagram of an application environment according to an exemplary embodiment;

[0086] Figure 2 is a flowchart of a method for generating a music video according to an exemplary embodiment;

[0087] Figure 3 is a flow chart of a method for acquiring a chord information set according to an exemplary embodiment;

[0088] Figure 4 is a flow chart of a method for acquiring a rhythm information set according to an exemplary embodiment;

[0089] Figure 5 is a schematic diagram of a training method for a lyrics encoder according to an exemplary embodiment;

[0090] Figure 6 is a flowchart of a method for generating an initial picture set according to an exemplary embodiment;

[0091] Figure 7 is a flow chart of a method for determining a semantic information index according to an exemplary embodiment;

[0092] Figure 8is a flow chart of a method for determining a semantic information index according to an exemplary embodiment;

[0093] Fig. 9 is a block diagram of an object semantic recognition device according to an exemplary embodiment;

[0094] Fig.10 It is a block diagram of an electronic device for music video generation or audio recognition according to an exemplary embodiment. DETAILED DESCRIPTION

[0095] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0096] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar first objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0097] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0098] See also Figure 1 , Figure 1 is a schematic diagram of an application environment of a music video generation method according to an exemplary embodiment. Figure 1 As shown, the application environment may include a server 01 and a client 02 .

[0099] In some possible embodiments, the server 01 may include an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud music and video generation, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The operating system running on the server may include but is not limited to Android, IOS, Linux, Windows, Unix, etc.

[0100] In some possible embodiments, the client 02 may include but is not limited to a smartphone, a desktop computer, a tablet computer, a laptop computer, a smart speaker, a digital assistant, an augmented reality (AR) / virtual reality (VR) device, a smart wearable device, etc. It may also be software running on the client, such as an application, a small program, etc. Optionally, the operating system running on the client may include but is not limited to Android system, IOS system, Linux, Windows, Unix, etc.

[0101] In some possible embodiments, the server 01 or the client 02 may obtain a lyrics sequence, a chord information set, and a rhythm information set of the target music, determine a lyrics semantic information sequence based on the lyrics sequence, the number of lyrics semantic information in the lyrics semantic information sequence is the same as the number of lyrics in the lyrics sequence, generate an initial picture set based on the lyrics semantic information sequence and the rhythm information set, perform style conversion on the pictures in the initial picture set using the chord information set, obtain a target picture set, determine a target video based on the target picture set and the target music, and the target video includes the target music. The present application obtains more pictures that fit the text and rhythm through the combination of music rhythm, chords, and text, and on this basis, further performs style conversion on the pictures using chord information to obtain video pictures with richer information and that fit the music better.

[0102] In some possible embodiments, the client 02 may be connected to the server 01 via a wired link or a wireless link.

[0103] In an exemplary embodiment, the client, the server, and the database corresponding to the server can all be node devices in the blockchain system, and can share the acquired and generated information with other node devices in the blockchain system to achieve information sharing between multiple node devices. Multiple node devices in the blockchain system can be configured with the same blockchain, which is composed of multiple blocks, and the adjacent blocks have an association relationship, so that when the data in any block is tampered, it can be detected by the next block, thereby preventing the data in the blockchain from being tampered with, and ensuring the security and reliability of the data in the blockchain.

[0104] Figure 2 is a flowchart of a method for generating a music video according to an exemplary embodiment. Figure 2 As shown, the music video generation method can be applied to a client, such as a mobile client, or to other node devices, such as a server. The method includes the following steps:

[0105] In step S201, the lyrics sequence, chord information set and rhythm information set of the target music are obtained.

[0106] In an embodiment of the present application, before obtaining the lyrics sequence, chord information set and rhythm information set of the target music on the device side, the target music can be obtained first.

[0107] Optionally, the target music can be a complete song, some song segments in some complete songs, or a combination of some song segments in some complete songs. Optionally, the target music can also be a song randomly generated by a user. In the embodiment of the present application, the generation of the target picture set is related to the semantic information of the lyrics. The above-mentioned complete song, some song segments in some complete songs, the combination of some song segments in some complete songs, and the song randomly generated by the user need to contain lyrics and cannot be pure music.

[0108] In an optional embodiment, obtaining the lyrics sequence of the target music includes: if the target music contains lyrics information, the lyrics information can be directly used as the lyrics sequence.

[0109] In another optional embodiment, if the target music does not contain lyrics information, the target music may be recognized using Automatic Speech Recognition (ASR) technology to obtain a lyrics sequence of the target music.

[0110] The lyrics sequence is a sequence of all lyrics sorted in the order of target music playback. In addition to the sorted lyrics, the lyrics sequence may also include the time information of each lyric. For example, assuming that the target music contains 10 lyrics, the lyrics sequence includes:

[0111] Sentence 1 "XXXXXXXX0", time information 1;

[0112] Sentence 2 "XXXXXXXX1", time information 2;

[0113] Sentence 3 "XXXXXXXX2", time information 3;

[0114] …

[0115] Sentence 10 "XXXXXXXX9", time information 10;

[0116] In an optional embodiment, the time information may be the timestamp information of each line of lyrics.

[0117] Optionally, the timestamp information is the start timestamp information of each sentence of lyrics. For example, the timestamp information of sentence 1 is 0 seconds, indicating that the start timestamp of sentence 1 is 0 seconds; the timestamp information of sentence 2 is 3 seconds, indicating that the start timestamp of sentence 2 is 3 seconds; the timestamp information of sentence 3 is 5.21 seconds, indicating that the start timestamp of sentence 3 is 5.21 seconds...

[0118] Optionally, the timestamp information is the end timestamp information of each sentence of lyrics. For example, if the timestamp information of sentence 1 is the 3rd second, it means that the end timestamp of sentence 1 is the 3rd second; if the timestamp information of sentence 2 is 5.21 seconds, it means that the end timestamp of sentence 2 is the 5.21th second; if the timestamp information of sentence 3 is 8 seconds, it means that the end timestamp of sentence 3 is the 8th second...

[0119] It can be understood that no matter whether the timestamp information is the start timestamp information or the end timestamp information, the device can determine the time period information of each sentence of lyrics through the timestamp information of the adjacent sentences of each sentence of lyrics, for example, the time period information of sentence 1 is 0-3 seconds, the time period information of sentence 2 is 3-5.21 seconds, and the time period information of sentence 3 is 5.21-8 seconds...

[0120] In another optional embodiment, the time information may be the time period information of each sentence of lyrics, for example, the time period information of sentence 1 is 0-3 seconds, the time period information of sentence 2 is 3-5.21 seconds, the time period information of sentence 3 is 5.21-8 seconds... Through the above time period information, the start timestamp information and end timestamp information of each sentence of lyrics can also be determined.

[0121] Figure 3 is a flow chart of a method for obtaining a chord information set according to an exemplary embodiment. Figure 3 As shown, including:

[0122] In step S301, the chords of the target music are analyzed to obtain a plurality of chord progression information arranged in the play order of the target music, and a time period of a music clip corresponding to each chord progression information in the target music.

[0123] In an embodiment of the present application, the device can analyze the chords of the target music to obtain multiple chord progression information arranged in the order of playing the target music, and the time period in the target music corresponding to each chord progression information.

[0124] It can be understood that the chord progression information can be determined based on the tonic that makes up the chord. For example, a C chord consists of notes 1, 3, and 5, that is, C, E, and G. The tonic is C, which is a first-degree chord.

[0125] According to the actual situation of the target music, optionally, one line of lyrics can correspond to multiple chord progression information arranged in the order of target music playback, such as the first line of lyrics corresponding to the first chord and the second chord. Optionally, one line of lyrics can correspond to one chord progression information, such as the second line of lyrics corresponding to the third chord.

[0126] Optionally, a chord progression information may span two lines of lyrics, for example, the first line of lyrics includes the first chord corresponding to the first part of the music and the second chord corresponding to the second part of the music, and the second line of lyrics includes the second chord corresponding to the first part of the music and the third chord corresponding to the second part of the music. It can be seen from this that the chord progression information of the second chord corresponds to the second part of the music of the first line of lyrics and the first part of the music of the second line of lyrics.

[0127] In order to enable the device to apply the chord progression information smoothly on a time basis later, when analyzing the chords of the target music, the device can determine the multiple chord progression information corresponding to the target music and the time period in the target music corresponding to each chord progression information. The multiple chord progression information is presented in the order in which the target music is played (or in chronological order).

[0128] Optionally, the chord progression information and the time period corresponding to the chord progression information in the target music may be represented based on each lyric:

[0129] Phrase 1 "XXXXXXXX0", 0-3 seconds; first chord, 0-2 seconds, second chord, 2-3 seconds;

[0130] Phrase 2 "XXXXXXXX1", 3-5.21 seconds; second-degree chord, 3-4 seconds, third-degree chord, 4-5.21 seconds;

[0131] Phrase 3 "XXXXXXXX2", 5.21-8 seconds; chord of degree three, 5.21-8 seconds;

[0132] …

[0133] Phrase 10 "XXXXXXXX9", 35.9-38.6 seconds; secondary chord, 35.9-38.6 seconds.

[0134] The target music contains 10 lines of lyrics, and the total time involved in the 10 lines of lyrics is 38.6 seconds. Taking the first line of lyrics as an example, the music clip corresponding to the first line of lyrics contains 2 chord progression information: first-order chords and second-order chords. The first chord progression information is a first-order chord, and the time period of the first-order chord in the target music is from the 0th second to the last 2 seconds, that is, the time period of the music clip corresponding to the first-order chord in the target music is from the 0th second to the last 2 seconds. The second chord progression information is a second-order chord, and the time period of the second-order chord in the target music is from the 2nd second to the last 3 seconds, that is, the time period of the music clip corresponding to the second-order chord in the target music is from the 2nd second to the last 3 seconds. Optionally, the chord progression information and the time period of the music clip corresponding to the chord progression information in the target music can be expressed based on the chord progression information:

[0135] First chord, 0-2 seconds; (including the first part of phrase 1)

[0136] Secondary chord, 2nd to 4th seconds; (including the latter part of phrase 1 and the former part of phrase 2)

[0137] Third degree chord, 4-8 seconds; (including the latter part of phrase 2 and phrase 3)

[0138] …

[0139] Secondary chord, 35.9-38.6 seconds; (including phrase 10).

[0140] That is to say, multiple chord progression information and the time period of the music clip corresponding to each chord progression information in the target music can be represented in multiple forms, so that subsequent steps can apply the chord progression information based on the chord progression information and the time period of the music clip corresponding to each chord progression information in the target music.

[0141] In step S303, a plurality of style information corresponding to a plurality of chord progression information is determined.

[0142] In the embodiment of the present application, the style information can be obtained from the device. Optionally, the device can obtain a correspondence table of chord technical information and style information from its own storage area, and the correspondence table stores style information expressed in words, for example, the style information corresponding to the first-level chord is "fresh", the style information corresponding to the second-level chord is "black and white", and the style information corresponding to the third-level chord is "literary"...

[0143] Optionally, different style information can be represented by a different picture. Each chord progression information can correspond to a style picture, for example, the first chord corresponds to picture A with style information of "fresh", the second chord corresponds to picture B with style information of "black and white", the third chord corresponds to picture C with style information of "literary"...

[0144] Optionally, different style information can be represented by different picture sets. Each chord progression information can correspond to one or more style picture sets. For example, the first-level chord corresponds to the picture set AA with the style information of "fresh", the second-level chord corresponds to the picture set BB with the style information of "black and white", the third-level chord corresponds to the picture set CC with the style information of "literary" and the picture set DD with the style information of "fashion"... Optionally, each picture set can include one or more pictures.

[0145] In step S305, a chord information set of the target music is determined based on the style information corresponding to each chord progression information and the time period of the music segment corresponding to each chord progression information in the target music.

[0146] Continuing to elaborate on the above, if the style information is expressed in text, then after obtaining the style information expressed in text, the device side can obtain the picture corresponding to the style information expressed in text. In the case where different style information is represented by a different picture, when the device side determines that the chord degree information is a first-level chord, it can obtain a picture A with the style information corresponding to the first-level chord being "fresh". If different style information can be represented by different picture sets, then when the device side determines that the chord degree information is a third-level chord, the device side can randomly determine a picture set from the picture set CC with the style information being "literary" and the picture set DD with the style information being "fashionable" or according to preset rules, such as the picture set CC of "literary" pictures. Subsequently, the device side can randomly determine another picture from the picture set CC or according to preset rules.

[0147] In this way, the device can determine the chord information set of the target music based on the style information corresponding to each chord progression information and the time period of the music clip corresponding to each chord progression information in the target music. The target music includes the music clip corresponding to each chord progression information.

[0148] Here is an example:

[0149] "Fresh" picture A, 0-2 seconds; (including the first part of the music in sentence 1)

[0150] "Black and white" picture B, 2-4 seconds; (including the latter part of the music of sentence 1 and the former part of the music of sentence 2)

[0151] Picture C of "Literary" (including the latter part of the music in sentence 2 and sentence 3)

[0152] …

[0153] "Black and white" picture B, 35.9-38.6 seconds; (including sentence 10).

[0154] Based on the above implementation method, the device side can determine the chord information set of the target music to prepare for the subsequent style conversion of the image, so that the video picture in the generated target video has a correlation with the chord information of the target music in terms of picture style, thereby improving the fit between the video and the target music.

[0155] Figure 4 is a flow chart of a method for acquiring a rhythm information set according to an exemplary embodiment. Figure 4 As shown, including:

[0156] In step S401, spectrum conversion is performed on the target music to obtain the mel spectrum corresponding to the target music.

[0157] In the embodiment of the present application, the device end can perform spectrum conversion on the target music to obtain the Mel spectrum corresponding to the target music. The Mel spectrum is a spectrum in Mel format, which is obtained by dot multiplication of the spectrum and several Mel filters. Specifically, the server can define a filter group with M filters, and the filter used is a triangular filter. The filter group of M filters can smooth the spectrum and eliminate the effect of harmonics, highlighting the resonance peak of the first semantic frequency domain information corresponding to each first sample semantic.

[0158] In step S403, the mel spectrum corresponding to the target music is subjected to impact separation processing to obtain a rhythm spectrum and a chord spectrum; the rhythm spectrum represents the regular changes in the rhythm of the target music, and the chord spectrum represents the chord melody of the target music.

[0159] In the embodiment of the present application, the device can perform impulse separation processing on the Mel spectrum corresponding to the target music to obtain a rhythm spectrum that emphasizes the rhythm and a chord spectrum that emphasizes the relationship between chords and melody. The rhythm spectrum represents the regular changes in the rhythm of the target music, and the chord spectrum represents the chord melody of the target music.

[0160] In step S405, amplitude variation is extracted from the rhythm spectrum to obtain amplitude variation data of the target music.

[0161] In step S407, momentum change extraction is performed on the chord spectrum to obtain momentum change data of the target music.

[0162] In the embodiment of the present application, the device can extract the amplitude change of the rhythm spectrum to obtain the amplitude change data of the target music. Optionally, the amplitude change data can be called average amplitude data. The formula of the amplitude change data of the target music is as follows:

[0163] Δa plus =a plus *spec_plus i ...Formula (1)

[0164] Among them, a plus is the impulse noise coefficient, which is obtained by normalizing the average value of the separated rhythm spectrum.

[0165] In the embodiment of the present application, the device can extract the momentum change of the chord spectrum to obtain the momentum change data of the target music. Optionally, the momentum change data can be called momentum direction smoothing data. The formula of the momentum change data of the target music is as follows:

[0166] Δa motion =motion*spec_motion i *motion_direct i oni*P(d)……Formula (2)

[0167] Wherein, d ranges from -1 to 1, including -1 and 1; a motion is the momentum noise coefficient, and its momentum direction is randomly resampled each time the chord spectrum switches.

[0168] Further: For the changes in the continuous difference, there are the following updates:

[0169] Δa plus =plus_noise i-1 ×s+Δa plus *(1-s)

[0170] Δa motion =motion_noise i1 *s+Δa motion *(1-s)……Formula (3)

[0171] In step S409, a rhythm information set is determined based on the amplitude change data and momentum change data of the target music; the rhythm information set includes multiple rhythm information and a timestamp of each rhythm information in the target music, and each rhythm information represents a frame energy value of the target music.

[0172] In an embodiment of the present application, the device side can determine a rhythm information set based on the amplitude change data and momentum change data of the target music; the rhythm information set includes multiple rhythm information and the timestamp of each rhythm information in the target music, wherein each rhythm information represents the frame energy value of the target music.

[0173] In an optional embodiment, the rhythm information set determined by the device may include the true energy value corresponding to each frame in the target music, that is, the rhythm information, wherein the dynamic information may be used to form the control noise, wherein the formula for controlling the noise is:

[0174]

[0175] The final fine-tuned control noise includes the previous noise smoothing and rhythm reconnection response, and the result of the accumulated momentum of the harmony. If the momentum accumulation data exceeds a certain range, it is necessary to re-initialize the reverse accumulation to ensure that the noise is within a reasonable range.

[0176]

[0177] In another optional embodiment, each of the multiple rhythm information included in the rhythm information set determined by the device based on the amplitude change data and momentum change data of the target music can exceed the preset information, and each rhythm information can represent the energy value of the rhythm information in the corresponding music clip that exceeds the preset value. For example, a certain rhythm information can be represented as the energy value of a frame corresponding to the first second in the music clip corresponding to the first line of lyrics. Specifically, the rhythm information can represent where the drum beats appear.

[0178] In this way, the device can determine the rhythm information set of the target music, and prepare for the subsequent generation of continuous pictures of each line of lyrics, so that the video images in the generated target video are correlated with the rhythm information of the target music in terms of picture conversion speed, thereby improving the fit between the video and the target music.

[0179] In step S203, a lyrics semantic information sequence is determined based on the lyrics sequence; the number of lyrics semantic information in the lyrics semantic information sequence is the same as the number of lyrics in the lyrics sequence.

[0180] In an embodiment of the present application, the device can determine the lyrics semantic information sequence based on the lyrics sequence. Specifically, the device can input each line of lyrics into a lyrics encoder, use the lyrics encoder to perform semantic analysis on each line of lyrics, and obtain lyrics semantic information corresponding to each line of lyrics.

[0181] The following introduces a training method for lyrics encoder based on contrastive learning. Figure 5FIG. 1 is a schematic diagram of a method for training a lyrics encoder according to an exemplary embodiment. Figure 5 As shown in the figure, during the training phase, a large number of paired text-images can be prepared. For example, if the text is "the earth's surface is bumpy", the image corresponding to the text is a picture of the moon with bumps on its surface. If the text is "a big tree with lush branches and leaves", the image corresponding to the text is a picture of a big tree.

[0182] During the training process, assuming that there are 1,000 pairs of text-images, the text "the bumpy surface of the earth" can be taken as a positive text example, that is, text 1, and the corresponding image is a picture of the moon with bumps on its surface as a positive image example, that is, image 1. The other 999 texts are taken as negative text examples, and the other 999 images are taken as negative image examples.

[0183] When you start training, Figure 5 The four encoders in (momentum text encoder, text encoder, image encoder, and momentum image encoder) can be untrained. Since a text is represented by a text semantic through a text encoder, and an image is represented by an image semantic through an image encoder, the device can input text 1 into the text encoder to obtain text semantics T, input the other 999 texts into the momentum text encoder to obtain 999 text semantics T1-TN, input image 1 into the image encoder to obtain image semantics I, and input the other 999 images into the momentum image encoder to obtain image semantics I1-IN.

[0184] The device can perform dot multiplication of the image semantics I corresponding to image 1 and the text semantics T and the text semantics T1-TN to obtain the loss corresponding to each text semantics, i.e., lossI2T. The text semantics T corresponding to text 1 and the image semantics I and the image semantics I1-IN can be performed dot multiplication to obtain the loss corresponding to each image semantics, i.e., lossT2I. Since text 1 and image 1 are a positive sample pair, while image 1 and other texts are negative samples, text 1 and other images are also negative samples. The subsequent training is to match the text semantics obtained by the text encoder of text 1 with the image semantics obtained by the image encoder of image 1. Theoretically, after spatial mapping, the two semantics should be the same. Therefore, the final training should be to minimize the distance (gap) between positive samples and maximize the distance (gap) between negative samples. The two losses can then be added to obtain the total loss, and the total loss can be used to update the parameters of the four encoders (momentum text encoder, text encoder, image encoder, and momentum image encoder).

[0185] Among them, the parameters of the text encoder and the picture encoder are updated according to the gradient calculated by back propagation. The parameters of the text encoder and the picture encoder are expressed as follows:

[0186]

[0187] in, Represents the gradient.

[0188] The parameter updates of the momentum text encoder and momentum image encoder do not depend on the gradient, but change according to the parameter updates of the positive example encoder. The momentum image encoder is updated as slowly as possible to avoid unstable updates of losses. The parameter formula of the momentum image encoder is as follows:

[0189]

[0190] Similarly, the momentum text encoder update parameters are expressed as follows:

[0191]

[0192] Among them, m in the above two formulas should be as close to 1 as possible to make the momentum image encoder update slowly.

[0193] Among them, the picture semantics I and the text semantics T, as well as the text semantics T1-TN are dot-multiplied to obtain the loss corresponding to each text semantic, that is, the formula of lossI2T is:

[0194]

[0195] in, represents a positive example of text, represents a negative example of text, represents the picture, and τ represents the thermal function.

[0196] The text semantics T and image semantics I corresponding to text 1, as well as the image semantics I1-IN, are dot-multiplied to obtain the loss corresponding to each image semantic, that is, the formula for lossT2I is:

[0197]

[0198] Then we get the global loss, that is, the formula for the total loss:

[0199] L total =L I2T +L T2I ...Formula (11)

[0200] In this way, the device can obtain the trained momentum text encoder, text encoder, image encoder and momentum image encoder by continuously training the above four encoders, and finally use the text encoder as a lyrics encoder.

[0201] Among them, the text semantics obtained after encoding text 1 by the trained text encoder and the image semantics obtained after encoding image 1 by the trained image encoder are almost the same.

[0202] According to the above description, the device can input each line of lyrics in the lyrics sequence into the trained text encoder to obtain the lyrics semantic information corresponding to each line of lyrics. Subsequently, the device can sort the lyrics semantic information corresponding to each line of lyrics according to the playback order of the lyrics in the target music to obtain a lyrics semantic information sequence. In other words, the number of lyrics semantic information in the lyrics semantic information sequence is the same as the number of lyrics in the lyrics sequence.

[0203] In step S205, an initial picture set is generated according to the lyrics semantic information sequence and the rhythm information set.

[0204] Figure 6 is a flowchart of a method for generating an initial picture set according to an exemplary embodiment, wherein the following operations are performed for each lyric semantic information in the lyric semantic information sequence, such as Figure 6 As shown, including:

[0205] In step S601, the time period of the lyrics corresponding to the lyrics semantic information being processed in the target music is determined.

[0206] In an optional implementation, the device side may perform the same operation on each lyric semantic information, which is explained below using the lyric semantic information being processed as an example.

[0207] In an optional implementation, the device can determine the time period of the lyrics corresponding to the lyrics semantic information being processed in the target music. Assuming that the lyrics corresponding to the lyrics semantic information being processed are the first lyrics, the time period of the first lyrics in the target music is 0-3 seconds.

[0208] In step S603, based on the time period of the lyrics corresponding to the lyrics semantic information in the target music, target rhythm information associated with the lyrics corresponding to the lyrics semantic information and a timestamp corresponding to the target rhythm information are determined from the rhythm information set.

[0209] In an optional implementation, the device can determine the target rhythm information associated with the lyrics corresponding to the lyrics semantic information and the timestamp corresponding to the target rhythm dynamic information from the rhythm information set based on the time period of the lyrics corresponding to the lyrics semantic information in the target music.

[0210] Next, we continue to analyze based on the first line of lyrics. Assuming that it is determined from the rhythm information set that the target rhythm information of the first line of lyrics from 0 to 3 seconds is 15% at the first second and 15% at the second second, that is, strong rhythmic drum beats appear at the first second and the second second, respectively. The frame energy values ​​corresponding to the first second and the second second are large, and each occupies 15% of the 100% energy in a line of lyrics. In this way, the device can obtain the target rhythm information associated with the first line of lyrics and the timestamp of the target rhythm information as follows: 15%, first second (or 1 / 3 time period); 15%, second second (or 2 / 3 time period).

[0211] In step S605, the first picture set corresponding to the lyrics semantic information is determined based on the lyrics semantic information, the target rhythm information, the timestamp corresponding to the target rhythm information, the time period of the lyrics corresponding to the lyrics semantic information in the target music and the preset picture frame rate.

[0212] In an optional implementation, the device end may determine the number of pictures in the first picture set corresponding to the lyrics semantic information based on the time period of the lyrics corresponding to the lyrics semantic information in the target music and the preset picture frame rate. Specifically, since the time period of the first line of lyrics is 3 seconds and the preset picture frame rate is 24 frames per second, the device end may determine the first line of lyrics, or the number of pictures in the first picture set corresponding to the current lyrics semantic information corresponding to the first line of lyrics, as the product of the time period of the lyrics corresponding to the current lyrics semantic information in the target music and the preset picture frame rate, that is, 72 pictures.

[0213] Optionally, the device can determine the semantic information index corresponding to each picture in the first picture set according to the target rhythm information, the timestamp corresponding to the target rhythm information, and the number of pictures in the first picture set. Based on the above paragraph, since the first second corresponds to the 24th picture of the 72 pictures (the target rhythm information is 15%), and the second second corresponds to the 48th picture of the 72 pictures (the target rhythm information is 15%), the device can determine that the total energy of the remaining 70 pictures is 70%, so that each of the remaining 70 pictures can be allocated 1% of the energy.

[0214] In order to make the lyrics semantic information reflect the coherence of adjacent pictures in the first picture set corresponding to the lyrics semantic information, and at the same time reflect the control of noise (rhythm information) in the first picture set, the device can determine the semantic information index corresponding to each picture in the first picture set containing 72 pictures based on the energy corresponding to each picture:

[0215] The semantic information index of picture 1 is 1, the semantic information index of picture 2 is 0.99, the semantic information index of picture 3 is 0.98... The semantic information index of picture 23 is 0.78, the semantic information index of picture 24 is 0.63, the semantic information index of picture 25 is 0.62... The semantic information index of picture 47 is 0.40, the semantic information index of picture 48 is 0.25, the semantic information index of picture 49 is 0.24... The semantic information index of picture 71 is 0.02, and the semantic information index of picture 72 is 0.01. In other words, except that the difference between the semantic information index of picture 23 and the semantic information index of picture 24 is only 0.15, and the difference between the voice information index of picture 47 and the semantic information index of picture 48 is only 0.15, the difference between the semantic information index of any other two pictures is arithmetic, with an interval of 0.01, which is consistent with the energy allocated to each picture. The semantic information index is used to characterize the change speed between the pictures finally presented in the first picture set based on the semantic information of the lyrics.

[0216] In an embodiment of the present application, the device end can generate the first picture set based on the semantic information index corresponding to each picture in the first picture set and the semantic information of the lyrics. Specifically, the device end can determine the semantic information corresponding to each picture in the first picture set based on the product of the semantic information index corresponding to each picture in the first picture set and the semantic information of the lyrics. Assuming that the current lyrics semantic information is F, the semantic information of picture 1 is 1F, the semantic information of picture 2 is 0.99F, the semantic information of picture 3 is 0.98F... The semantic information of picture 23 is 0.78F, the semantic information of picture 24 is 0.63F, and the semantic information of picture 25 is 0.62F... The voice information of picture 47 is 0.40F, the semantic information of picture 48 is 0.25F, and the semantic information of picture 49 is 24F... The semantic information of picture 71 is 0.02F, and the voice information of picture 72 is 0.01F. In this way, the device end can obtain the semantic information corresponding to each picture in the first picture set.

[0217] Subsequently, the device can use the image decoder to decode the semantic information corresponding to each image in the first image set, and obtain the first image set corresponding to each lyric semantic information. Since the generation of each image in the first image set is based on the lyric semantic information, and in two adjacent images with an energy difference of 0.01 (such as the first and second images), the semantic information is less different, so the screen presentation of the two adjacent images finally generated is also less different. However, in two adjacent images with an energy difference of 0.15 (such as the 23rd and 24th images), since the semantic information is relatively different, the screen presentation of the two images finally generated is also relatively different.

[0218] In the above embodiment, the gradually decreasing semantic information index of the pictures in the first picture set can not only make the different pictures in the first picture set maintain the continuity between the pictures while showing obvious changes, but also reflect the rapid changes of the pictures when the rhythm is strong through the rhythm information, so that the rhythm between the picture changes and the music is more closely matched.

[0219] In another process of generating the first picture set, the device not only uses the current lyrics voice information, but also uses the adjacent lyrics semantic information to increase the correlation between the first picture set corresponding to the current lyrics and the first picture set corresponding to the adjacent lyrics, and reduce the abruptness of the sudden transition between lyrics and pictures.

[0220] Figure 7 is a flow chart of a method for determining a semantic information index according to an exemplary embodiment. Figure 7 As shown, including:

[0221] In step S701, a first semantic information index and a second semantic information index corresponding to each picture in the first picture set are determined according to the target rhythm information, the timestamp corresponding to the target rhythm information and the number of pictures in the first picture set; the first semantic information index corresponds to the lyrics semantic information; the second semantic information index corresponds to the adjacent lyrics voice information of the lyrics semantic information; the first semantic information index and the second semantic information index comprehensively represent the picture screen change speed of each picture in the first picture set.

[0222] Optionally, the device may determine a first semantic information index and a second semantic information index corresponding to each picture in the first picture set according to the target rhythm information, a timestamp corresponding to the target rhythm information, and the number of pictures in the first picture set.

[0223] Based on the elaboration in the previous paragraph, since the first second corresponds to the 24th picture among the 72 pictures (the target rhythm information is 15%), and the second second corresponds to the 48th picture among the 72 pictures (the target rhythm information is 15%), the device can determine that the total energy of the remaining 70 pictures is 70%. In this way, each of the remaining 70 pictures can be allocated 1% of the energy.

[0224] In order to make the lyrics semantic information reflect the coherence of the adjacent pictures in the first picture set corresponding to the lyrics semantic information, reflect the presentation of the controlled noise (rhythm information) in the first picture set, and establish coherence between adjacent lyrics, the device can determine the first semantic information index and the second semantic information index corresponding to each picture in the first picture set of 72 pictures, where the first semantic information index corresponds to the lyrics semantic information; the second semantic information index corresponds to the adjacent lyrics voice information of the lyrics semantic information; the first semantic information index and the second semantic information index comprehensively represent the picture screen change speed of each picture in the first picture set:

[0225] The first semantic information index of picture 1 is 1, and the second semantic information index is 0.01;

[0226] The first semantic information index of picture 2 is 0.99, and the second semantic information index is 0.02;

[0227] The first semantic information index of picture 3 is 0.98, and the second semantic information index is 0.03;

[0228] …

[0229] The first semantic information index of picture 23 is 0.78, and the second semantic information index is 0.23;

[0230] The first semantic information index of picture 24 is 0.63, and the second semantic information index is 0.38;

[0231] The first semantic information index of picture 25 is 0.62, and the second semantic information index is 0.39;

[0232] …

[0233] The first semantic information index of picture 47 is 0.40, and the second semantic information index is 0.61;

[0234] The first semantic information index of picture 48 is 0.25, and the second semantic information index is 0.76;

[0235] The first semantic information index of image 49 is 0.24, and the second semantic information index is 0.77;

[0236] …

[0237] The first semantic information index of picture 71 is 0.02, and the second semantic information index is 0.99;

[0238] The first semantic information index of picture 72 is 0.01, and the second semantic information index is 1.

[0239] In step S703, the first semantic information corresponding to each picture is determined based on the first semantic information index corresponding to each picture and the lyrics semantic information.

[0240] In the embodiment of the present application, assuming that the lyrics semantic information is F, the device end can determine the first semantic information corresponding to each picture based on the product of the first semantic information index corresponding to each picture and the lyrics semantic information;

[0241] Among them, the first semantic information corresponding to each picture is:

[0242] The first semantic information index of picture 1 is 1F,

[0243] The first semantic information index of picture 2 is 0.99F.

[0244] The first semantic information of picture 3 is 0.98F.

[0245] …

[0246] The first semantic information of picture 23 is 0.78F.

[0247] The first semantic information of picture 24 is 0.63F.

[0248] The first semantic information of picture 25 is 0.62F.

[0249] …

[0250] The first semantic information of picture 47 is 0.40F.

[0251] The first semantic information of picture 48 is 0.25F.

[0252] The first semantic information of picture 49 is 0.24F.

[0253] …

[0254] The first semantic information of picture 71 is 0.02F.

[0255] The first semantic information of picture 72 is 0.01F.

[0256] In step S705, the second semantic information corresponding to each picture is determined based on the second semantic information index corresponding to each picture and the adjacent lyrics semantic information.

[0257] In the embodiment of the present application, assuming that the adjacent lyrics semantic information is G, the device end can calculate the second semantic information corresponding to each picture based on the product of the second semantic information index corresponding to each picture and the adjacent lyrics semantic information;

[0258] The second semantic information index of picture 1 is 0.01G;

[0259] The second semantic information index of picture 2 is 0.02G;

[0260] The second semantic information index of picture 3 is 0.03G;

[0261] …

[0262] The second semantic information index of picture 23 is 0.23G;

[0263] The second semantic information index of picture 24 is 0.38G;

[0264] The second semantic information index of picture 25 is 0.39G;

[0265] …

[0266] The second semantic information index of picture 47 is 0.61G;

[0267] The second semantic information index of picture 48 is 0.76G;

[0268] The second semantic information index of picture 49 is 0.77G;

[0269] …

[0270] The second semantic information index of picture 71 is 0.99G;

[0271] The second semantic information index of picture 72 is 1G.

[0272] In step S707, comprehensive semantic information corresponding to each picture is determined based on the first semantic information and the second semantic information corresponding to each picture.

[0273] In the embodiment of the present application, the device can determine the comprehensive semantic information corresponding to each picture based on the sum of the first semantic information and the second semantic information corresponding to each picture. For example, the comprehensive semantic information of picture 1 is F+0.01G, and the comprehensive semantic information of picture 72 is 0.01F+G.

[0274] In step S709, a first picture set is generated based on the comprehensive semantic information corresponding to each picture; the adjacent lyrics semantic information is the previous lyrics semantic information or the next lyrics semantic information of the lyrics semantic information in the lyrics semantic information sequence.

[0275] Subsequently, the device may use a picture decoder to decode the comprehensive semantic information corresponding to each picture in the first picture set to obtain the first picture set corresponding to each lyric semantic information.

[0276] In an optional embodiment, if the adjacent lyrics semantic information is the next lyrics semantic information, then except for the last lyrics semantic information, the adjacent lyrics semantic information of each lyrics semantic information is the next lyrics semantic information of each lyrics semantic information, and the adjacent lyrics semantic information of the last lyrics semantic information is a null value.

[0277] In another optional embodiment, if the adjacent lyrics semantic information is the previous lyrics voice information, then except for the first lyrics semantic information, the adjacent lyrics semantic information of each lyrics semantic information is the previous lyrics semantic information of each lyrics semantic information, and the adjacent lyrics semantic information of the first lyrics semantic information is a null value.

[0278] In the above-mentioned embodiment, the pictures in the first picture set can not only maintain the coherence between the different pictures in the first picture set while obviously changing through the gradually decreasing first semantic information index and the gradually increasing second semantic information index, but also further reflect the excessive coherence of the pictures corresponding to adjacent lyrics. At the same time, the rapid changes of the pictures during strong rhythms are reflected through the rhythm information, so that the rhythm between the picture changes and the music is more closely matched. In step S607, when the first picture set corresponding to each lyric semantic information is determined, the first picture set corresponding to each lyric semantic information is arranged in a preset order to obtain an initial picture set; the preset order is the time sequence of the lyrics corresponding to each lyric semantic information in the target music.

[0279] Optionally, the device can arrange the first picture sets corresponding to each lyric semantic information in a preset order to obtain an initial picture set. The preset order is the time sequence of the lyrics corresponding to each lyric semantic information in the target music. That is, the initial picture set contains multiple first picture sets, and each first picture set is sorted according to the playing time of the corresponding music clip in the target music.

[0280] Based on the above embodiment, the device can combine the lyrics generation of the target music and the pictures mapped with the lyrics, and use the rhythm of the target music to modify the changes between pictures and the rules of changes, so that the obtained initial picture set and the music are more closely matched.

[0281] In an optional implementation, the device side can generate an initial picture set based on the lyrics semantic information sequence and the rhythm information set. Specifically, the device side can input each lyric semantic information and the rhythm information set corresponding to the lyric semantic information into the picture generation network, and use the picture generation network to decode to obtain the first picture set corresponding to each lyric semantic information. Subsequently, the first picture set corresponding to each lyric semantic information constitutes the initial picture set corresponding to the entire target music.

[0282] Figure 8is a schematic diagram of a picture generation network according to an exemplary embodiment. Figure 8 It is specifically introduced how to train the image generation network so that the trained image generation network can be subsequently applied to the generation of images in the first image set.

[0283] In an optional embodiment, Figure 8 The text encoder in can be Figure 5 The trained text encoder in Figure 8 The picture encoder in can be Figure 5 The trained image encoder in . First, the device can input the training text into a text editor to obtain embedded feature information embedding 1, which is a feature of the cross-modal space. However, due to the large semantic space, the image generated by the completely random initial latent space may be seriously distorted. Therefore, a fully connected layer is designed to map the embedded feature information embedding 1 to a limited number of code encodings through the dictionary codebook to obtain code 1.

[0284] Next, the device can input encoding 1 as a parameter into a mapping network composed of several fully connected layers to decouple the input latent space encoding 1 parameters to form a more interpretable feature.

[0285] In an optional embodiment, if Figure 8 As shown in the figure, in addition to inputting the control noise and features into the image generation network, you can also input an initial constant C1. Combined with the input of the features, you can scale the features in the variance dimension and adjust the convolution ratio. The scaling method is as follows:

[0286]

[0287] Among them, w represents the weight parameter. This parameter has a certain degree of randomness, and the feature map will be scaled to a certain extent after scaling. si represents the proportional value of the input feature. That is, after the lyrics semantic information is given, the time points of the lyrics semantic information and the adjacent lyrics semantic information are determined, and the feature maps fa and fb are obtained respectively. The difference between the two feature vectors is obtained with the frame number difference, and the intermediate process Si is obtained. Si is used as the feature input image generation network, where the formula of Si is as follows:

[0288] s i =lerp(f a , f b , t), t∈n frame ...Formula (13)

[0289] Subsequently, the generated image is obtained through upsampling and restoration of the image generation network. Optionally, the device side can determine whether the generated image is a real image. In an optional embodiment, the device side can perform feature sampling on the training image set through a real image feature sampler to obtain the features of the real image. Optionally, each image in the image training set is obtained by taking an image acquisition device. The features of the real image can be color features, relationship features between objects on the image, proportion features of objects on the image, etc.

[0290] Optionally, after obtaining the features of the real image, the device can perform authenticity discrimination on the generated image based on the features of the real image to obtain a loss value, and use the loss value to reversely train the image generation network so that the features in the image generated by the image generation network are closer and closer to the features of the real image, so that the discriminator cannot determine whether the generated image is real or synthetic. The formula for the loss value obtained by authenticity discrimination is:

[0291] (14)

[0293] Furthermore, in order to make the generated image restore the text semantics as much as possible in terms of image semantics, the device can input the generated network into the image encoder to obtain embedded feature information embedding 2. Subsequently, the device maps the embedded feature information embedding 2 to a limited number of code codes through the dictionary Codebook to obtain code 2.

[0294] Optionally, the device side can compare the embedded feature information embedding1 and the embedded feature information embedding2 to obtain the feature loss, and compare the loss of encoding 1 and encoding 2 to obtain the encoding loss. Subsequently, the device side can update the image generation network based on the feature loss and the encoding loss, and after a preset number of trainings, a trained image generation network can be obtained.

[0295] In step S207, the chord information set is used to perform style conversion on the pictures in the initial picture set to obtain a target picture set.

[0296] Optionally, the device can determine the target style information corresponding to each first picture set and the time period corresponding to the target style information in the initial picture set based on the time period of each line of lyrics in the target music, the style information corresponding to each chord progression information in the chord information set, and the time period of each chord progression information in the target music; the first picture set is the picture set corresponding to each lyric semantic information in the initial picture set.

[0297] Based on the target style information and the time period corresponding to the target style information, style conversion is performed on each picture in each first picture set to obtain a second picture set corresponding to each lyric semantic information; the pictures in the second picture set correspond one-to-one to the pictures in the first picture set, and the picture contents of the two one-to-one corresponding pictures are the same but the picture styles are different.

[0298] Subsequently, the device end may arrange the second picture sets (10 second picture sets corresponding to the 10 first picture sets) in the order of playing the target music to obtain the target music.

[0299] For example, the second picture set corresponding to the first line of lyrics includes 72 pictures. Since the first chord corresponding to the first part of the first part of the first line of the lyrics at 0-2 seconds corresponds to the picture A with a "fresh" style, and the first chord corresponding to the second part of the second part of the lyrics (including the first part of the lyrics) at 2-3 seconds corresponds to the picture B with a "black and white" style, the second picture set finally includes the first 48 pictures with a "fresh" style and the last 24 pictures with a "black and white" style.

[0300] In step S209, a target video is determined based on the target picture set and the target music; the target video includes the target music.

[0301] In the embodiment of the present application, the device end can align the target picture set and the target music to determine the target audio and video to obtain the target audio and video. That is, the device end can align the target picture set and the target music, perform splicing, and obtain the target video. As the target video is played, the pictures in the target picture set are displayed frame by frame. A video that has a relationship with the lyrics semantics, rhythm, and chords of the target music is obtained.

[0302] Fig. 9 is a block diagram of a music video generation device according to an exemplary embodiment. The device has the function of implementing the data processing method in the above method embodiment, and the function can be implemented by hardware or by hardware executing corresponding software. Fig. 9 The device includes an information acquisition module 901, an information sequence acquisition module 902, a picture set generation module 903, a style conversion module 904 and a video determination module 905:

[0303] An information acquisition module is configured to acquire a lyrics sequence, a chord information set, and a rhythm information set of a target music;

[0304] The information sequence acquisition module is configured to determine the lyrics semantic information sequence based on the lyrics sequence; the number of lyrics semantic information in the lyrics semantic information sequence is the same as the number of lyrics in the lyrics sequence;

[0305] A picture set generation module is configured to generate an initial picture set according to the lyrics semantic information sequence and the rhythm information set;

[0306] A style conversion module is configured to perform style conversion on the pictures in the initial picture set using the chord information set to obtain a target picture set;

[0307] The video determination module is configured to determine a target video based on a target picture set and target music; the target video includes target music.

[0308] In some possible embodiments, the information acquisition module is configured to execute:

[0309] Analyze the chords of the target music to obtain a plurality of chord progression information arranged in the order of playing the target music, and a time period of a music clip corresponding to each chord progression information in the target music;

[0310] Determine a plurality of style information corresponding to a plurality of chord progression information one by one;

[0311] The chord information set of the target music is determined based on the style information corresponding to each chord progression information and the time period of the music segment corresponding to each chord progression information in the target music.

[0312] In some possible embodiments, the information acquisition module is configured to execute:

[0313] Perform spectrum conversion on the target music to obtain the Mel spectrum corresponding to the target music;

[0314] The mel spectrum corresponding to the target music is subjected to impact separation processing to obtain a rhythm spectrum and a chord spectrum; the rhythm spectrum represents the regular changes in the rhythm of the target music, and the chord spectrum represents the chord melody of the target music;

[0315] Extract the amplitude variation of the rhythm spectrum to obtain the amplitude variation data of the target music;

[0316] Extract momentum changes from the chord spectrum to obtain momentum change data of the target music;

[0317] A rhythm information set is determined based on the amplitude change data and momentum change data of the target music; the rhythm information set includes multiple rhythm information and a timestamp of each rhythm information in the target music; each rhythm information represents a frame energy value of the target music.

[0318] In some possible embodiments, the picture set generation module is configured to execute:

[0319] For each lyric semantic information in the lyric semantic information sequence, the following is performed: determining a time period of the lyrics corresponding to the lyric semantic information being processed in the target music;

[0320] Based on the time period of the lyrics corresponding to the lyrics semantic information in the target music, determining the target rhythm information associated with the lyrics corresponding to the lyrics semantic information and the timestamp corresponding to the target rhythm information from the rhythm information set;

[0321] Determine a first picture set corresponding to the lyrics semantic information based on the lyrics semantic information, the target rhythm information, the timestamp corresponding to the target rhythm information, the time period of the lyrics corresponding to the lyrics semantic information in the target music, and the preset picture frame rate;

[0322] When the first picture set corresponding to each lyric semantic information is determined, the first picture set corresponding to each lyric semantic information is arranged in a preset order to obtain an initial picture set; the preset order is the time sequence of the lyrics corresponding to each lyric semantic information in the target music.

[0323] In some possible embodiments, the picture set generation module is configured to execute:

[0324] Determine the number of pictures in the first picture set corresponding to the lyrics semantic information based on the time period of the lyrics corresponding to the lyrics semantic information in the target music and the preset picture frame rate;

[0325] Determine a semantic information index corresponding to each picture in the first picture set according to the target rhythm information, a timestamp corresponding to the target rhythm information, and the number of pictures in the first picture set; the semantic information index represents a picture screen change speed of each picture in the first picture set;

[0326] The first picture set is generated based on the semantic information index corresponding to each picture in the first picture set and the lyrics semantic information.

[0327] In some possible embodiments, the picture set generation module is configured to execute:

[0328] Determine a first semantic information index and a second semantic information index corresponding to each picture in the first picture set according to the target rhythm information, the timestamp corresponding to the target rhythm information, and the number of pictures in the first picture set; the first semantic information index corresponds to the lyrics semantic information; the second semantic information index corresponds to the adjacent lyrics voice information of the lyrics semantic information; the first semantic information index and the second semantic information index comprehensively represent the picture screen change speed of each picture in the first picture set;

[0329] Generating the first picture set based on the semantic information index corresponding to each picture in the first picture set and the lyrics semantic information includes:

[0330] Determine the first semantic information corresponding to each picture based on the first semantic information index corresponding to each picture and the lyrics semantic information;

[0331] Determine the second semantic information corresponding to each picture based on the second semantic information index corresponding to each picture and the semantic information of adjacent lyrics;

[0332] Determine comprehensive semantic information corresponding to each picture based on the first semantic information and the second semantic information corresponding to each picture;

[0333] Generate a first picture set based on comprehensive semantic information corresponding to each picture;

[0334] The adjacent lyrics semantic information is the previous lyrics semantic information or the next lyrics semantic information of the lyrics semantic information in the lyrics semantic information sequence.

[0335] In some possible embodiments, the style transfer module is configured to perform:

[0336] Based on the time period of each lyric in the target music, the style information corresponding to each chord progression information in the chord information set, and the time period of each chord progression information in the target music, determine the target style information corresponding to each first picture set in the initial picture set and the time period corresponding to the target style information; the first picture set is the picture set corresponding to each lyric semantic information in the initial picture set;

[0337] Performing style conversion on each picture in each first picture set based on the target style information and the time period corresponding to the target style information, to obtain a second picture set corresponding to each lyric semantic information; the pictures in the second picture set correspond to the pictures in the initial picture set one-to-one, and the two pictures that correspond one-to-one have the same picture content but different picture styles;

[0338] Arrange the second picture set in the order of playing the target music to obtain the target picture set.

[0339] It should be noted that the device provided in the above embodiment, when implementing its functions, is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0340] Fig.10 A block diagram of an apparatus 3000 for music video generation or audio recognition is shown according to an exemplary embodiment. For example, the apparatus 3000 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0341] Reference Fig.10, the device 3000 may include one or more of the following components: a processing component 3002, a memory 3004, a power component 3006, a multimedia component 3008, an audio component 3010, an input / output (I / O) interface 3012, a sensor component 3014, and a music video generation component 3016.

[0342] The processing component 3002 generally controls the overall operation of the device 3000, such as operations associated with display, phone calls, data music video generation, camera operation, and recording operation. The processing component 3002 may include one or more processors 3020 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 3002 may include one or more modules to facilitate the interaction between the processing component 3002 and other components. For example, the processing component 3002 may include a multimedia module to facilitate the interaction between the multimedia component 3008 and the processing component 3002.

[0343] The memory 3004 is configured to store various types of data to support operations on the device 3000. Examples of such data include instructions for any application or method operating on the device 3000, contact data, phone book data, messages, pictures, videos, etc. The memory 3004 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0344] The power supply component 3006 provides power to the various components of the device 3000. The power supply component 3006 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 3000.

[0345] The multimedia component 3008 includes a screen that provides an output interface between the device 3000 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 3008 includes a front camera and / or a rear camera. When the device 3000 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0346] The audio component 3010 is configured to output and / or input audio signals. For example, the audio component 3010 includes a microphone (MIC), and when the device 3000 is in an operation mode, such as a call mode, a recording mode, and a semantic recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 3004 or sent via the music video generation component 3016. In some embodiments, the audio component 3010 also includes a speaker for outputting audio signals.

[0347] I / O interface 3012 provides an interface between processing component 3002 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0348] The sensor assembly 3014 includes one or more sensors for providing various aspects of status assessment for the device 3000. For example, the sensor assembly 3014 can detect the open / closed state of the device 3000, the relative positioning of components, such as the display and keypad of the device 3000, and the sensor assembly 3014 can also detect changes in the position of the device 3000 or a component of the device 3000, the presence or absence of user contact with the device 3000, the orientation or acceleration / deceleration of the device 3000, and the temperature change of the device 3000. The sensor assembly 3014 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 3014 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 3014 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0349] The music video generation component 3016 is configured to facilitate wired or wireless music video generation between the device 3000 and other devices. The device 3000 can access a wireless network based on a music video generation standard, such as WiFi, an operator network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the music video generation component 3016 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the music video generation component 3016 also includes a near-field music video generation (NFC) module to facilitate short-range music video generation. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0350] In an exemplary embodiment, the apparatus 3000 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0351] An embodiment of the present invention also provides a computer-readable storage medium, which can be set in an electronic device to store at least one instruction or at least one program related to implementing a music video generation method. The at least one instruction or the at least one program is loaded and executed by the processor to implement the music video generation method provided by the above method embodiment.

[0352] An embodiment of the present invention further provides a computer program product, which includes a computer program, the computer program is stored in a readable storage medium, and at least one processor of a computer device reads and executes the computer program from the readable storage medium, so that the computer device performs any method in the first aspect of the embodiments of the present disclosure.

[0353] It should be noted that the sequence of the embodiments of the present invention described above is for description only and does not represent the advantages and disadvantages of the embodiments. The above describes specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0354] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0355] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0356] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A music video generation method, characterized in that: include: Obtaining a lyrics sequence, a chord information set, and a rhythm information set of the target music; Determine a lyrics semantic information sequence based on the lyrics sequence; The amount of lyrics semantic information in the lyrics semantic information sequence is the same as the number of lyrics in the lyrics sequence; Generate an initial picture set according to the lyrics semantic information sequence and the rhythm information set; Performing style conversion on pictures in the initial picture set using style information corresponding to the chord information set to obtain a target picture set; A target video is determined based on the target picture set and the target music; the target video includes the target music.

2. The music video generation method according to claim 1, characterized in that: The step of obtaining the chord information set of the target music includes: Analyze the chords of the target music to obtain a plurality of chord progression information arranged in the order of playing the target music, and a time period of a music clip corresponding to each chord progression information in the target music; Determine a plurality of style information corresponding to the plurality of chord progression information one by one; The chord information set of the target music is determined based on the style information corresponding to each chord progression information and the time period of the music clip corresponding to each chord progression information in the target music.

3. The music video generation method according to claim 1 or 2, characterized in that: The step of obtaining the rhythm information set of the target music includes: Performing spectrum conversion on the target music to obtain a mel spectrum corresponding to the target music; Performing impact separation processing on the mel spectrum corresponding to the target music to obtain a rhythm spectrum and a chord spectrum; the rhythm spectrum represents that the rhythm of the target music is changing regularly, and the chord spectrum represents the chord melody of the target music; Extracting amplitude changes of the rhythm spectrum to obtain amplitude change data of the target music; Extracting momentum changes of the chord spectrum to obtain momentum change data of the target music; A rhythm information set is determined based on the amplitude change data and momentum change data of the target music; the rhythm information set includes multiple rhythm information and a timestamp of each rhythm information in the target music; each rhythm information represents a frame energy value of the target music.

4. The music video generation method according to claim 3, characterized in that: The generating an initial picture set according to the lyrics semantic information sequence and the rhythm information set comprises: For each lyric semantic information in the lyric semantic information sequence, the following steps are performed: determining a time period in the target music of the lyrics corresponding to the lyric semantic information being processed; Based on the time period of the lyrics corresponding to the lyrics semantic information in the target music, determining the target rhythm information associated with the lyrics corresponding to the lyrics semantic information and the timestamp corresponding to the target rhythm information from the rhythm information set; Determine a first picture set corresponding to the lyrics semantic information based on the lyrics semantic information, the target rhythm information, a timestamp corresponding to the target rhythm information, a time period of the lyrics corresponding to the lyrics semantic information in the target music, and a preset picture frame rate; When the first picture set corresponding to each lyric semantic information is determined, the first picture set corresponding to each lyric semantic information is arranged in a preset order to obtain the initial picture set; the preset order is the chronological order of the lyrics corresponding to each lyric semantic information in the target music.

5. The music video generation method according to claim 4, characterized in that: The determining of the first picture set corresponding to the lyrics semantic information based on the lyrics semantic information, the target rhythm information, a timestamp corresponding to the target rhythm information, a time period of the lyrics corresponding to the lyrics semantic information in the target music, and a preset picture frame rate includes: Determining the number of pictures in the first picture set corresponding to the lyrics semantic information based on the time period of the lyrics corresponding to the lyrics semantic information in the target music and the preset picture frame rate; Determine a semantic information index corresponding to each picture in the first picture set according to the target rhythm information, a timestamp corresponding to the target rhythm information, and the number of pictures in the first picture set; the semantic information index represents a picture screen change speed of each picture in the first picture set; The first picture set is generated based on the semantic information index corresponding to each picture in the first picture set and the lyrics semantic information.

6. The music video generation method according to claim 5, characterized in that: The determining, according to the target rhythm information, a timestamp corresponding to the target rhythm information, and the number of pictures in the first picture set, a semantic information index corresponding to each picture in the first picture set includes: Determine a first semantic information index and a second semantic information index corresponding to each picture in the first picture set according to the target rhythm information, a timestamp corresponding to the target rhythm information, and the number of pictures in the first picture set; the first semantic information index corresponds to the lyrics semantic information; the second semantic information index corresponds to adjacent lyrics voice information of the lyrics semantic information; the first semantic information index and the second semantic information index comprehensively represent a picture screen change speed of each picture in the first picture set; The generating the first picture set based on the semantic information index corresponding to each picture in the first picture set and the lyrics semantic information includes: Determine the first semantic information corresponding to each picture based on the first semantic information index corresponding to each picture and the lyrics semantic information; Determining the second semantic information corresponding to each picture based on the second semantic information index corresponding to each picture and the adjacent lyrics semantic information; Determining comprehensive semantic information corresponding to each picture based on the first semantic information and the second semantic information corresponding to each picture; generating the first picture set based on the comprehensive semantic information corresponding to each picture; The adjacent lyrics semantic information is the previous lyrics semantic information or the next lyrics semantic information of the lyrics semantic information in the lyrics semantic information sequence.

7. The music video generation method according to claim 2, characterized in that: The step of performing style conversion on the pictures in the initial picture set by using the style information corresponding to the chord information set to obtain a target picture set includes: Based on the time period of each lyric in the target music, the style information corresponding to each chord progression information in the chord information set, and the time period of each chord progression information in the target music, determine the target style information corresponding to each first picture set in the initial picture set and the time period corresponding to the target style information; the first picture set is the picture set corresponding to each lyric semantic information in the initial picture set; Performing style conversion on each picture in each first picture set based on the target style information and the time period corresponding to the target style information, to obtain a second picture set corresponding to each lyric semantic information; the pictures in the second picture set correspond to the pictures in the initial picture set one-to-one, and the two pictures in one-to-one correspondence have the same picture content but different picture styles; Arrange the second picture set in the order of playing the target music to obtain the target picture set.

8. A music video generating device, characterized in that: include: An information acquisition module is configured to acquire a lyrics sequence, a chord information set, and a rhythm information set of a target music; An information sequence acquisition module is configured to determine a lyrics semantic information sequence based on the lyrics sequence; the number of lyrics semantic information in the lyrics semantic information sequence is the same as the number of lyrics in the lyrics sequence; A picture set generation module is configured to generate an initial picture set according to the lyrics semantic information sequence and the rhythm information set; A style conversion module is configured to perform style conversion on the pictures in the initial picture set using the style information corresponding to the chord information set to obtain a target picture set; The video determination module is configured to determine a target video based on the target picture set and the target music; the target video includes the target music.

9. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the music video generation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the music video generating method as described in any one of claims 1 to 7.

Images